mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-27 01:48:13 +00:00
60716facb405b6f60a2fd1d4b38e4b815febfa20
433
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dborup
|
60716facb4 |
fix: address MINOR notes on PR #1852's merge-ready review
[dijkstra] doc-noted that `total` in handleNodes' filtered path reflects only the returned page-slice, not an all-pages count — matches pre-existing semantics, not a regression, but worth being explicit that fetchAllNodes (public/app.js) never reads it for pagination (relies on page length instead). [munger] the pagination compensation loop's maxIterations=50 safety cap now logs a WARN breadcrumb when it's actually hit, instead of silently returning a possibly-short page. New test seeds 51 consecutive hidden-prefix nodes at limit=1 to force the cap and asserts both the empty result and the log line. |
||
|
dborup
|
81bc9930af |
fix: address bot review on PR #1852 (2 MAJOR races + cleanup)
MAJOR (carmack): handleConfigClient and nodeListPostFilters both read
s.cfg.GeoFilter unlocked, but the field is cfgMu-guarded — PUT
/api/config/geo-filter swaps it under Lock(). Both sites now go through
s.getGeoFilter() like the other existing read sites.
MAJOR (dijkstra): window.MC_GEO_FILTER is set asynchronously by roles.js's
/api/config/client fetch, but nodes.js read it synchronously during
loadNodes(). On a cold page load where the Nodes page renders before that
fetch resolves, every node was silently classified as domestic until the
user clicked a filter chip. loadNodes() now awaits window.MeshConfigReady
(which never rejects) before the geo-scope filter runs. New regression
test proves the race with a manually-controlled promise, and fails
against the pre-fix code (verified via git stash).
MINOR: stale test-file header comment (still described the filter as
using the `foreign` flag, predating 6012ed0's lat/lon rewrite); loose
interface{} != float64 comparison in the geoFilter config test replaced
with explicit type assertions so a shape change fails loudly instead of
comparing unequal to the wrong type.
NIT: trimmed an 8-line struct-field doc comment; added
role="group"/aria-label to both the Status and Domestic/Foreign
filter-button groups on the Nodes page (bot noted Status already lacked
one too).
|
||
|
dborup
|
e3692b4642 |
fix: /api/nodes pagination truncated whenever a post-LIMIT filter removed a row
geo_filter, node blacklist, and hidden-name-prefix were all applied to a page AFTER the SQL LIMIT already fixed its size, so a page that was genuinely full at the DB layer could come back shorter than requested. fetchAllNodes (public/app.js) treats "page shorter than requested" as "this was the last page" and stops — so a single filtered-out row anywhere in a page silently truncated everything after it. Found on stg.meshview.dk: the live map showed ~450 of ~1300 GPS-valid nodes because a hidden-name-prefix node happened to land in the first 500-row page. handleNodes now loops, re-fetching and re-filtering additional DB pages, until it has collected a full page of post-filter results or the DB itself runs dry (a raw page shorter than requested = genuine end of data) — restoring the "short page really means end of data" contract fetchAllNodes relies on. Regression test reproduces the exact scenario and fails against the old code (verified via git stash). |
||
|
dborup
|
2c80b44787 |
fix: Domestic/Foreign node filter now classifies from lat/lon, not the stale foreign flag
The `foreign` flag only reflects nodes whose ADVERT was decoded and classified AFTER geo_filter was configured. On stg.meshview.dk this made the just-shipped filter nearly useless: only 27 of 1503 nodes were flagged foreign, while 570 actually have GPS placing them outside the configured box (543 of those simply never got re-flagged). Exposes the geo_filter box/polygon via /api/config/client (new GeoFilter field on ClientConfigResponse), and adds nodePassesGeoFilter + helpers to app.js — a faithful JS port of internal/geofilter's PassesFilter/PointInPolygon/DistToSegmentKm, cross-checked line-for-line against the real Go implementation for bbox, polygon, and buffer cases. Nodes with no GPS fix (or (0,0)) count as domestic, matching both the Go semantics and the user's explicit choice for this feature. nodes.js's Domestic/Foreign filter-group (added in f99df27) now calls nodePassesGeoFilter(n.lat, n.lon, window.MC_GEO_FILTER) instead of reading n.foreign. |
||
|
dborup
|
865bf50155 |
docs: correct overstated cardinality claim on GetChannelScopeAdoption
Bot review on PR #1852 (munger, MAJOR-adjacent→MINOR): "bounded by the 1-byte channel hash space" only holds for encrypted channels (256 enc_%02x buckets) — plain-text CHAN channels use a free-form name string (ingestor/db.go), so the bound isn't as hard as the comment claimed. Correctness of the uncap itself is unaffected; this is a doc-only fix. |
||
|
dborup
|
d947ada49a |
feat: show all channels in Scope Adoption, filter encrypted/unencrypted
GetChannelScopeAdoption was capped at the top 30 channels by message volume, silently hiding quieter channels from the Scopes tab. Channel cardinality is bounded by the 1-byte hash space, so the cap wasn't protecting against anything real — removed it. Client gets a filter (All/Unencrypted/Encrypted) based on the 'enc_' channel_hash prefix db.go already uses for channels the ingestor couldn't decrypt, so a visitor can separate plain channels from encrypted ones without hunting through a long unfiltered list. |
||
|
dborupandClaude Sonnet 5
|
7141acca7a |
fix: address remaining bot-review findings on the geo_filter/Foreign Traffic work
- MAJOR: config.example.json was missing geoFilterExemptNodeList entirely — every other Config field in this repo gets a sibling _comment_<field> entry (established, actively-used convention throughout the file, ~20 existing examples), documented here to match. Also corrected the adjacent geo_filter _comment, which predates this PR and claimed it "restricts ingestion" — it doesn't; that's foreignAdverts.mode=drop's job, unrelated to this flag. - MINOR: ?geoFilter=<v> only recognized the literal string "1" to enable and treated anything else (including "true") as disabling — a real footgun since "true" reads as an obviously valid boolean value. Now accepts 1/true and 0/false as explicit overrides; any other value (including absent) falls through to the deployment default instead of silently flipping it off. - Tightened the Foreign Traffic tab's stop-hook test: it previously only asserted stop() doesn't throw, which a no-op stub would also pass. Sandbox's setInterval/clearInterval are now real spies tracking live/cleared ids, so the test verifies render() registers exactly one interval and stop() actually clears that exact id (idempotently on a second call). - NIT: relays.sort now wraps unscoped_relay_count_24h in Number(...) defensively (AGENTS.md § Type Safety — cast at the boundary). - NIT: foreignNodes.sort now parses last_seen once per node (decorate-sort-undecorate) instead of twice per comparator call. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
dborupandClaude Sonnet 5
|
b255dceaa8 |
fix: flip geo_filter node-list default polarity — safe-by-default for upgraders
7cc5aae's GeoFilterAppliesToNodeList had the polarity backwards: it defaulted to false, meaning geo_filter would NOT apply to the node list unless a deployment explicitly opted in — but every existing deployment that already had geo_filter configured (predating this field entirely) would decode that field as false too, since it's simply absent from their config.json. They'd silently get the new "show everything" behavior on upgrade with no way back short of reading release notes and adding a new config key — exactly the breaking change the opt-in gate was meant to prevent, just one layer deeper. Renamed to GeoFilterExemptNodeList (still defaults false): geo_filter now applies to the node list by DEFAULT when configured, matching the long-standing #730 behavior exactly, for every deployment predating this field. Only a deployment that explicitly sets geoFilterExemptNodeList=true (i.e. one adopting geo_filter fresh, purely for foreign_advert classification/analytics) gets the non-filtering default. ?geoFilter=0/1 still overrides either default for a single request. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
dborupandClaude Sonnet 5
|
7cc5aae6ac |
fix: add GeoFilterAppliesToNodeList escape hatch for the geoFilter opt-in flip
Per bot review on PR #1852 (comment 5012514227): the ?geoFilter=1
opt-in change (
|
||
|
dborupandClaude Sonnet 5
|
ffb2c842af |
fix: make handleNodes' geo_filter exclusion opt-in (?geoFilter=1)
Configuring geo_filter alone was silently changing what GET /api/nodes returned — every node outside the polygon and not yet foreign_advert -tagged (which only happens on that node's next ADVERT after geo_filter is set) vanished from every view built on this endpoint, including the live map. There was no way to see them again short of waiting for each one to re-advertise. geo_filter's own purposes — the ingestor tagging foreign adverts, and the explicit prune-geo-filter admin flow — are untouched; this only gates the passive node-list declutter view behind an explicit query param, so turning geo_filter on to build analytics (e.g. the Foreign Traffic tab) doesn't have the side effect of hiding nodes from the map. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
dborupandClaude Sonnet 5
|
10eed4b2f6 |
fix: address remaining bot-review findings on PR #1852
Working through the outstanding non-blocker findings across all five
automated review passes on the PR:
- MAJOR: GetChannelMessages' rows.Scan() error was silently discarded
— a schema mismatch would produce zero-valued messages instead of a
visible error. Now returns the error, matching sibling query loops.
- Extracted the hashRegions name-normalization rule (trim, "#"-prefix,
dedupe) shared between cmd/ingestor's loadRegionKeys and cmd/server's
region-utilization diff into a new internal/regions package, so the
two can no longer drift apart on the rule independently.
- Batched GetRepeaterNamesByKeys' SQL IN (...) clause in chunks of 500
— an unbounded clause risks SQLITE_MAX_VARIABLE_NUMBER on very large
deployments' byPathHop candidate sets.
- handleScopeStats' remaining silently-swallowed err==nil branches
(GetMatchedRegionNames, GetNodesByDefaultScope,
GetChannelMessageScopeStats, GetChannelScopeAdoption) now log a WARN
breadcrumb on failure instead of failing invisibly.
- Scope Adoption table was rendering 3 of the 4 ChannelScopeAdoption
fields (Unscoped omitted) — the visible numbers didn't reconcile to
the message total without doing the subtraction by hand. Added the
column.
- Removed a duplicate `vertical-align: middle` declaration on
.badge-transport (dead, not a behavior change).
- Deleted ChannelMessageResp — flagged for interface{} vs *string/*int
typing, but turned out to be completely unused dead code; removing
it resolves the finding more directly than retyping something
nothing constructs.
Left two NIT/MINOR items as-is with reasoning:
- renderRegionNodeGroups' inline styles match the same pattern used in
15+ other places in analytics.js — "fixing" only this one function
would make it less consistent with the file, not more.
- TransmissionResp's interface{} fields (a different struct than the
one just removed) are an established, actively-used pattern for
nullable SQL-scanned values across that whole response type;
retyping it is a much larger, unrelated refactor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
dborup
|
b3c3b0d17a |
feat: SIGHUP reloads hashChannels/hashRegions without a restart
channelKeys and regionKeys (derived from config.json's hashChannels/
hashRegions) were loaded exactly once in main() and held in plain
maps for the process lifetime — adding a new hashtag channel or
region required a full ingestor restart to take effect, which also
resets the in-memory relay/scope-attribution state built up since the
last restart (see the Repeaters-by-Region cold-start discussion).
Wraps both maps in atomic.Pointer-backed hotKeys (same pattern already
used for Config.NodeBlacklist, just actually wired to a trigger this
time) and adds a SIGHUP handler that re-reads config.json and
atomically swaps in freshly-derived keys:
kill -HUP <ingestor-pid>
On a malformed config during reload, the previous keys are kept and
the error is logged — a bad edit must not blank out a working
ingestor. The MQTT message-handler closure now pulls a fresh snapshot
from the atomic pointers per message instead of closing over a static
map, so no other call sites needed to change.
|
||
|
dborup
|
064a9c8e35 |
feat: scope adoption per channel on the Scopes analytics tab
Adds channelScopeAdoption to /api/scope-stats: the existing ChannelMessages aggregate (scoped/unscoped/unknown for channel chat) broken down PER CHANNEL — which specific channels (#test, #wardriving, ...) actually use region scoping vs which never do. Ordered by message volume, capped at the top 30 channels. Frontend renders a compact table under "Channel Messages": channel name, total messages, scoped count+%, unknown count. |
||
|
dborup
|
03abeba00e |
feat: hour-of-day activity heatmap on the Scopes analytics tab
Adds hourlyActivityByRegion to /api/scope-stats: each region's message counts bucketed by hour-of-day (0-23 UTC), aggregated across every day in the window — answers "when during a typical day is this region active" rather than "how did volume change over the window" (that's the existing chronological TimeSeries chart). Frontend renders a compact heatmap: one row per region, 24 hour columns, color intensity normalized per-row (each region's own busiest hour) so a quiet region's daily shape stays visible next to a loud one instead of being crushed toward zero. |
||
|
dborup
|
1aed3ee5c8 |
feat: bridge-repeaters breakdown on the Scopes analytics tab
Adds bridgeRepeaters to /api/scope-stats: RepeatersByRegion inverted into pubkey -> regions, keeping only repeaters that have relayed traffic for MORE than one region. These are the mesh's literal backbone nodes connecting otherwise-separate regional communities — losing one is a more consequential failure than losing a single-region repeater. Computed inline while building RepeatersByRegion (reuses the same byRegion map and role-filtered names lookup, no extra queries). Frontend renders a small table under "Repeaters by Region": repeater name (linked to its node detail page), region count, and the region list. |
||
|
dborup
|
c74a005247 |
feat: channel-messages-only scoped/unscoped breakdown on the Scopes tab
Adds channelMessages to /api/scope-stats: the same scoped/unscoped/ unknown question as the main Summary, but restricted to payload_type=5 (channel chat) instead of all observed traffic. Most channel chat is plain FLOOD rather than transport-scoped, so this can read very differently from the all-traffic numbers — answers "how many of our actual channel messages carry a region scope" directly instead of requiring the reader to infer it from the broader stats. New GetChannelMessageScopeStats() mirrors GetScopeStats' query shape but scopes TotalMessages to ALL route types for payload_type=5 (not just route_type 0/3), since restricting to transport routes would answer a different question than "how many channel messages, period". Frontend renders a small "Channel Messages" stat-card row under the main summary cards, window-scoped like the rest of the tab. |
||
|
dborup
|
ea340554e9 |
fix: don't credit TransportedScopes from the ambiguous prefix bucket
Root-caused via a real report: "Repeaters by Region" showed a hyper-
local scope (#dk-fyn-middelfart) as transported by repeaters spread
across the whole country, which shouldn't be possible — MeshCore
firmware only relays a TRANSPORT_FLOOD/DIRECT packet when the
repeater's OWN configured region matches the packet's transport code
(examples/simple_repeater/MyMesh.cpp allowPacketForward, gated by
RegionMap::findMatch against the repeater's local region list).
Confirmed against the meshcore-dev/MeshCore source.
The bug: both TransportedScopes computation paths (bulk
computeRepeaterRelayInfoMap and per-node GetRepeaterRelayInfo) fold a
full pubkey's byPathHop entries together with its matching 1-byte
raw-prefix bucket — an intentional, existing fallback for RelayCount/
LastRelayed ("this node is probably active") that tolerates the
1-byte hash's inherent ambiguity (any node sharing that first byte
gets folded in). Applying the SAME fold to TransportedScopes asserted
something far more specific than the ambiguous signal can support,
and something the protocol itself wouldn't allow.
Scope accumulation now only happens on the exact-key (resolved,
unambiguous) pass; RelayCount/LastRelayed/RelayActive keep the
prefix-bucket fold unchanged, since those remain intentionally
approximate. Added relayEntry.fromPrefix to carry this distinction
through the per-node path, and rewrote the test that had pinned the
old (incorrect) folding behavior.
|
||
|
dborup
|
a44f2a4ad5 |
feat: nodes-running-this-region breakdown on the Scopes analytics tab
Adds originatingNodesByRegion to /api/scope-stats: nodes whose OWN default_scope (#899) is a given region, complementing the existing repeatersByRegion (transported_scopes) breakdown. The distinction matters — a repeater can relay traffic for a region it isn't itself configured with, so "who runs this region" and "who has carried this region's traffic" are different, both useful questions. Frontend: refactored the per-region collapsible-list rendering (used by both breakdowns) into a shared renderRegionNodeGroups() helper instead of duplicating the HTML-building logic, and added a "Nodes Running This Region" section alongside "Repeaters by Region". |
||
|
dborup
|
93144ee654 |
fix: exclude byPathHop bucket keys from repeaters-by-region
byPathHop indexes both full pubkeys and short hex-prefix "bucket" keys
used internally for ambiguous-hop resolution — GetRepeaterRelayInfoMap
returns TransportedScopes for every one of those keys indiscriminately.
The first deploy of repeatersByRegion iterated the raw map and fell
back to showing the bucket key itself when no name matched, so a
handful of short internal keys were counted as "repeaters" (stg showed
594 "repeaters" for #dk — every active repeater plus every 2-6 char
bucket key that ever touched a #dk packet).
GetNodeNamesByKeys is now GetRepeaterNamesByKeys and filters
`role IN ('repeater','room')` in the SQL itself, so a key only survives
if it's a real node — bucket keys never match a nodes.public_key row
and are dropped rather than falling back to the raw key.
|
||
|
dborup
|
87d2f479a7 |
feat: repeaters-by-region breakdown on the Scopes analytics tab
Adds repeatersByRegion to /api/scope-stats: for every region that has ever matched a transmission, which distinct repeaters/rooms have relayed traffic carrying that scope. Sourced from the same 5-min background-recomputed bulk relay-info cache the Nodes page already uses (GetRepeaterRelayInfoMap / TransportedScopes, #1751) — no new expensive computation, just an inversion + name lookup. Frontend renders a collapsible per-region repeater list (name links to the node detail page) under a new "Repeaters by Region" section, explicitly framed as a coverage/redundancy signal: a region carried by only one repeater is a single point of failure for that area. |
||
|
dborup
|
afa76c4f52 |
feat: region-utilization report on the Scopes analytics tab
Adds configuredRegions/unusedRegions to /api/scope-stats: an all-time (not window-scoped) diff between the operator's configured hashRegions list and the set of scope_name values that have actually matched a transmission still in retention. Surfaces how much of the region list is dead weight — directly actionable evidence for pruning, which is also the real fix for the HMAC-collision noise (fewer configured regions -> lower birthday-collision probability per packet). Server config now parses hashRegions (previously ingestor-only, same config.json key) purely to read the configured names — no HMAC key derivation happens server-side. Frontend: a "Region Utilization" section on the Scopes tab shows used/unused counts and a collapsible list of the unused region names. |
||
|
dborup
|
aaef080f94 |
feat: surface scope on the Packets tab's transport badge
The packet detail pane already showed scope (with an "unknown scope" fallback for empty scope_name), but the packets table itself gave no at-a-glance signal — the existing transportBadge() "T" marker only encoded route type. transportBadge() now takes an optional scopeName argument: a resolved region enriches the tooltip, an empty scope_name (transport-eligible but unmatched/ambiguous) renders as "T?" with a distinct muted badge style instead of the confident amber, so it's visually distinguishable from a resolved scope without relying on color alone. Existing callers that don't pass scopeName (live.js) are unaffected. QueryGroupedPackets (SQLite + in-memory) didn't select scope_name at all — added it, since the Packets tab defaults to the grouped view. |
||
|
dborup
|
a94d57ed4a |
feat: show "Scope: unknown" for transport-scoped messages with no match
GetChannelMessages already returned an empty scope string for transport-eligible packets whose region couldn't be determined (no configured region matched, or matchScope now reports an HMAC collision as unknown), but the UI treated empty scope the same as "not applicable" and rendered nothing — indistinguishable from a plain FLOOD/DIRECT message that never carries a scope at all. Adds route_type to the channel-message payload (both SQLite and in-memory paths, plus the decrypt-candidate and live WS paths) so the frontend can tell "not transport-scoped" (routeType 1/2, no tag) apart from "transport-scoped but unresolved" (routeType 0/3 with empty scope, now shown as "Scope: unknown"). |
||
|
dborup
|
8765de71e0 |
fix: matchScope reports unknown instead of guessing on HMAC collision
transport_code_1 is only 16 bits, so with enough configured hashRegions
a different, unrelated region's HMAC can coincidentally also match a
packet's real code1 (expected rate ~= len(regionKeys)/65534 per
packet). matchScope used to return the first match found while
iterating the region-key map, whose order Go randomizes per process —
so an ambiguous packet's assigned scope was effectively a coin flip
that could even change across ingestor restarts.
Confirmed on stg.meshview.dk (1098 configured regions): a live GRP_TXT
message's code1=C417 matched both "#dk" (the sender's true region) and
"#dk1906" (an unrelated collider), and was observed resolving to
either name across sends.
matchScope now scans every configured region and only returns a name
when exactly one matches; two or more matches are reported as
unknown-scoped ("") rather than guessed, with a log line for operator
visibility. Doesn't fix the root collision-probability cause (that
needs fewer/coarser configured regions) but stops confidently
displaying a wrong single answer.
|
||
|
dborup
|
350bf7eeee |
fix: surface scope_name on live WS channel-message broadcast
The WS packet broadcast (IngestNewFromDB / IngestNewObservations in cmd/server/store.go) builds its own map independent of the REST response helpers, so scope_name was missing there even after it was added to GetChannelMessages and txToMap. Adds it to both broadcast paths and wires channels.js's live-append handler to read it, so brand-new messages show their scope immediately instead of waiting for the next periodic REST refresh. |
||
|
dborup
|
c686ae3f37 |
feat: surface transport scope on channel messages
Adds scope_name to GetChannelMessages (both SQLite and in-memory store paths) and to the general packet response shape, then renders it as "Scope: <name>" in the channel chat meta line so operators can see which region scope a message was transported under. |
||
|
Kpa-clawbotandcorescope-bot
|
59cf5130d1 |
fix(1838): fold non-transport routes into scope-stats Unscoped (#1842)
Fixes #1838 ## Problem `/api/scope-stats` reported 100% scoped whenever any region was configured. Reporter noticed on a scopeless instance that "unscoped" was always zero — the pie visual is misleading to operators deciding on `denyf *`. ## Root cause `cmd/server/db.go:22` restricted the entire scope-stats denominator to `route_type IN (0, 3)`. Per firmware `docs/packet_format.md § Route Types`: - `0` = `TRANSPORT_FLOOD` - `1` = `FLOOD` - `2` = `DIRECT` - `3` = `TRANSPORT_DIRECT` Only routes 0 and 3 carry `transport_code_1` (transport-level scope). Routes 1 and 2 are inherently unscoped by protocol. The existing SQL was correct for the "how many transport-scopable routes are actually scoped" question, but the denominator was silently promoted to "all traffic" in the UI. Bonus: the comment on `routeTypeTransportSQL` labelled routes 0+3 as "FLOOD (0) and DIRECT (3)" — wrong on both counts. ## Fix - `cmd/server/db.go` — corrected the `routeTypeTransportSQL` comment; added `routeTypeNonTransportSQL = "route_type IN (1, 2)"` alongside it. - `GetScopeStats` runs a second `COUNT(*)` over `route_type IN (1,2) AND first_seen >= ?` and folds that count into `Summary.Unscoped`. Same index path as the existing query — one extra scan per `/api/scope-stats` call (cached 30s per triage's carmack finding). - `public/analytics.js` — Scopes tab header explains the denominator (all observed transmissions) and which route types carry scope. Card notes now render `X% of all traffic` for Scoped/Unscoped and `X% of scoped` for Unknown Scope so the pie's denominator is explicit. ## TDD - Red: `5554ffe4` — extended `TestGetScopeStats` + `TestHandleScopeStats` with `route_type=1` and `route_type=2` rows and asserted `Unscoped = 3` (1 transport-NULL + 2 non-transport). Ran the tests and confirmed assertion failure (`Unscoped = 1, want 3`). - Green: `ebbb9253` — implementation + label copy. Full `go test ./cmd/server/...` passes (54s). ## Preflight overrides - check-branch-clean: justified — cross-stack fix by design (backend semantics change + matching frontend label copy). All 4 files are exactly the surface the triage comment identified. ## Verification - `go test ./cmd/server/...` — 54s, all pass. - Firmware confirmation: `firmware/docs/packet_format.md:20-24` (route type table). ## Files touched - `cmd/server/db.go` — comment fix + second COUNT query. - `cmd/server/db_test.go` — extended fixture. - `cmd/server/routes_test.go` — extended fixture + isolate from seed data. - `public/analytics.js` — labels and header copy. --------- Co-authored-by: corescope-bot <bot@corescope.dev> |
||
|
|
d60188e481 |
refactor(1828): split handleObserverAnalytics into 5 helpers + byTxID fast-path (#1839)
## Summary Phase A of #1828: extract the 5 aggregate builders in `handleObserverAnalytics` into pure helpers in a new `cmd/server/observer_analytics.go`. Handler becomes a snapshot + filter + 5 composed calls. Also adopts the `byTxID` direct-read in `buildPacketTypes` (issue body's core observation): the payload-type histogram no longer allocates a full `enrichObs` map + interface-boxed fields just to read `tx.PayloadType`. That's the ~90% perf win the triage called out. Scope is exactly Phase A per the second triage comment. Phase B (sub-endpoints, caching, SQL migration) is deferred to a follow-up. ## Byte-identical output - Timeline / NodesTimeline: same key set, same sort, same labels. - PacketTypes: same keys/counts. Both legacy (`enriched["payload_type"].(int)`) and new (`tx.PayloadType == nil` guard) skip obs whose tx is missing or `PayloadType` is `nil`. - SnrDistribution: same 2-unit floor bucketing (negative-side rounding preserved), same ascending sort. - RecentPackets: still the first 20 enriched observations (`enrichObs` kept only here, where the extra fields are actually needed). ## TDD - Red commit: `9dc62f43` — 7 unit tests fail on assertions (not build errors) against stubs. - Green commit: `8d41011d` — implementations + handler rewire. All new tests + existing `TestObserverAnalytics*` handler tests pass. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → clean (all 8 hard gates + 3 warnings pass). ## Non-goals - No new endpoints. - No SQL migration. - No public API signature change. - Snapshot count unchanged (still one under RLock, per #1481 P0-2). Fixes #1828. --------- Co-authored-by: fix-1828-bot <bot@corescope.local> Co-authored-by: clawbot <bot@corescope> |
||
|
Michael J. ArcanandWaydroid Builder
|
d2ef624c2e |
feat(api): flood_advert_count_7d on the node detail endpoint (#1831)
Adds, per node, how many distinct FLOOD adverts it originated in the last 7 days. Zero-hop adverts (route_type DIRECT) are excluded, so a nearby observer hearing a node's cheap local adverts does not inflate the number - the existing advert_count mixes both kinds and cannot tell a chatty flooder (mesh-wide airtime) from the recommended 240-minute zero-hop cadence (local only). Consumers (the ArcScope repeater advisor) rate advert hygiene against the community practice of one flood advert every ~49h; with the mixed total, a correctly configured repeater looked chatty whenever an observer sat within zero-hop range. Implemented like the relay-liveness fields: a pure, unit-tested counter over (first_seen, route_type, hash) entries with the same timestamp parsing and hash dedup, fed by a from_pubkey-indexed query capped at the 2000 most recent advert rows. The flood route-type constant is named advertRouteTypeFlood so this merges independently of the open unscoped-relay PR (#1823). --------- Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
56fe844871 |
test: dedupe the unscoped-relay tests via a shared fixture (#1832)
Follow-up to #1823: TestRepeaterUnscopedRelayCount and its _Bulk twin were ~30 verbatim lines apart (DB, node insert, store seeding, assertions), differing only in the lookup under test - seeding changes had to land twice. Both now use a shared seedUnscopedRelayFixture + assertUnscopedCounts and contain only their respective lookup call. No behaviour change; the relay-liveness suite passes. Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
bd0a58e14c |
feat(api): add unscoped_relay_count_24h per-node field (#1823)
## What Adds a per-node API field `unscoped_relay_count_24h` on repeater/room nodes: the number of the node's 24h relay-hops that were unscoped floods (route_type == ROUTE_TYPE_FLOOD). A strict subset of relay_count_24h. ## Why A well-configured repeater runs `flood.max.unscoped 0` and should not rebroadcast unscoped floods — each one is re-sent by every repeater that hears it, so one packet turns into mesh-wide traffic. Exposing this lets clients (the ArcScope repeater advisor) detect and flag that base-config problem from observed packets. ## How Computed like relay_count_24h in both paths (bulk /api/nodes + per-node detail) with a route_type==FLOOD filter; reuses the byPathHop index, no migration. Wired into both handlers + OpenAPI schema + unit tests (per-node and bulk). Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
|
096e16409c |
fix(#1741): wrap test-DB insert loops in a single transaction (#1819)
## Fixes #1741 `TestBoundedLoad_OldestLoadedSet` (and any test building a 5000-row fixture) hung/timed out, blocking reliable `go test ./cmd/server` and CI. ## Root cause The four test-DB builders in `cmd/server/bounded_load_test.go` (`createTestDBAt`, `createTestDBWithObs`, `createTestDBWithAgedPackets`) inserted rows in a loop with no `BEGIN`/`COMMIT`. With the pure-Go `modernc.org/sqlite` driver every `Exec` auto-commits → one fsync per row → ~2N fsyncs for N transmissions (tx + obs). At `numTx=5000` that's ~10k fsyncs and the fixture blows past the test timeout. Sibling tests with `numTx<=3000` happened to stay under the timeout, so only the 5000-row cases visibly hung. ## Fix Wrap each insert loop in a single `BEGIN`/`COMMIT` so the whole fixture build becomes one commit. Fixtures now finish in well under a second regardless of `numTx`; the tests' actual assertions (`oldestLoaded` set, newest-first ordering, bounded load) are exercised instead of the timeout masking them. Also made the prepared-statement `Exec` calls check their error (previously discarded) so a failed insert surfaces instead of silently leaving the DB short. No production code changed — test infrastructure only. ## Verified - `TestBoundedLoad_OldestLoadedSet`: **0.18s** (was: 30s timeout / FAIL). - Full `TestBoundedLoad*` + retention group: passes in ~1.2s. - `go test ./...` in `cmd/server`: exit 0 (no longer blocks on this test). Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
6a32ec2b2d |
fix(#1729): preserve firmware-default Public channel (0x11) in analytics (#1817)
## Fixes #1729 The firmware-default **Public** channel (channel-hash byte `0x11` = 17) was rendered as an opaque **"Encrypted (0x11)"** row at the bottom of the analytics Channels tab, despite the key being well-known and builtin. ## Root cause `computeAnalyticsChannels` applied the #978 rainbow-table validation (`SHA256(SHA256("#name")[:16])[0]`, the **hashtag** hash scheme) to every decoded channel name. The Public channel is a **PSK** channel whose hash byte is key-derived (`SHA256(key)[0]` = 17), not hashtag-derived (`186` for `#Public`). So the ingestor-decoded name `"Public"` failed the hashtag check and was discarded, the row forced to `encrypted=true, name="ch17"`. ## Fix Trust the ingestor's `decryptionStatus`. The ingestor already persists `decryptionStatus:"decrypted"` when it decoded a packet with a real key (PSK), and `"no_key"` / `"decryption_failed"` otherwise. When the packet is `decrypted`, skip the hashtag hash check and keep the name — it came from a key-based decryption, not a rainbow-table lookup. The #978 mismatch rejection still applies to non-decrypted packets, so rainbow-table collisions are still caught. Frontend needs no change: `encrypted=false, name="Public"` lands in the "Network" group (top), not "Encrypted". ## Tests - `makeGrpTx` gains `makeGrpTxWithStatus` companion to set `decryptionStatus`. - `TestComputeAnalyticsChannels_PublicChannelPreserved`: hash 17 / "Public" / `decrypted` → name stays `"Public"`, `encrypted=false`. - `TestComputeAnalyticsChannels_UndecryptedNameStillValidated`: a non-`decrypted` name failing the hashtag check is still downgraded to `ch17` (#978 regression guard). All channel-analytics tests pass; `go build ./...` clean. Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
750b8742a7 |
fix(staging-compose): decouple in-container mosquitto from standalone broker (#1813)
Red commit: `3898dbc5` (verified locally — CI run URL pending)
## Problem
A standalone `mqtt-broker` container (`eclipse-mosquitto:2`) was
provisioned out-of-band on the staging VM. It now owns MQTT, is attached
to external docker network `meshcore-net`, and binds host port `8883`.
The current `docker-compose.staging.yml` still:
- Publishes `1883:1883` on the host (dead weight; conflicts the moment
the broker moves to that port).
- Defaults `DISABLE_MOSQUITTO=false`, so the in-container mosquitto
burns RAM and briefly contests the `mqtt-broker` docker DNS name on cold
start.
- Doesn't join `meshcore-net`, so the ingestor can't resolve
`mqtt-broker:1883` via docker DNS without manual surgery.
## Fix (`docker-compose.staging.yml` only)
1. Remove the `1883:1883` host port publish from `staging-go`.
2. Flip `DISABLE_MOSQUITTO` default from `false` to `true`. Operators
can opt back in with the env var.
3. Attach `staging-go` to both `default` and `meshcore-net`; declare
`meshcore-net` as `external: true` so the file never tries to
create/destroy operator state.
Healthcheck and Caddy/443 plumbing untouched (out of scope).
## Test added (TDD framing: Option A — Go shape-asserts)
`cmd/server/staging_compose_broker_test.go:1` adds four regex-based
assertions on the compose file shape:
- staging-go does **not** bind port `1883` in ANY form (quoted/unquoted
short form, or long-form `target: 1883` / `published: 1883`).
- `DISABLE_MOSQUITTO` uses the interpolated default form
`${DISABLE_MOSQUITTO:-true}` (preserves operator override). Bare literal
`true`, or a later `=false` override in the same env block, is rejected.
- Top-level `networks:` declares `meshcore-net` as `external: true`.
- `staging-go` attaches to `meshcore-net` via a real
`services.staging-go.networks:` sub-key (comment-stripped so an
in-comment example can't masquerade).
Regex (not YAML byte-equality) so cosmetic edits don't break the guard.
No new go module deps. Red commit `3898dbc5` fails all 4 assertions on
master. Green commit `38297ff4` makes them pass. Round-1 hardening
commit `9f7155e2` tightens the regexes (per adversarial + kent-beck
must-fixes) and was verified against master's YAML shape — all 4 tests
fail on `origin/master`'s compose, pass on branch, proving the tightened
regexes still gate a real regression.
## Risk
Low, with one intentional semantic change.
- **Semantic change (v3.7+):** `DISABLE_MOSQUITTO` in
`docker-compose.staging.yml` now defaults to `true`. This is a
**deliberate flip** — the standalone `mqtt-broker` container is now
authoritative on the staging host, and running the in-container
mosquitto alongside it wastes RAM and races the docker DNS name
`mqtt-broker` on cold start. Operators who want the pre-v3.7 shape
(in-container mosquitto + host-published `1883`) must explicitly opt
back in via env override AND re-add the `1883:1883` port mapping
(concrete snippet is inline in the compose file and in `DEPLOY.md` under
"Standalone MQTT broker (staging)"). This intent is called out in a
`SEMANTIC CHANGE (v3.7+)` header comment at the top of
`docker-compose.staging.yml`.
- **Deploy prereq:** the external `meshcore-net` docker network MUST
already exist on the host before `docker compose up`. If it doesn't,
compose refuses to start `staging-go`. This is documented inline in the
compose file (with the `docker network create meshcore-net` one-liner)
and in `DEPLOY.md`.
- **Only takes effect where the standalone broker is deployed** — which
it already is on staging today. The legacy `DISABLE_MOSQUITTO=false`
path remains reachable via env override; the ingestor's upstream config
is untouched.
Partial fix — no tracking issue; follow-up to operator-side broker
provisioning.
---------
Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
|
||
|
|
fa15ab0a30 |
fix(#1809): gate background loader on LoadChunked completion (#1811)
Partial fix for #1809. Red commit: |
||
|
|
242c7c609b |
fix(mqtt): escalate persistent paho disconnect + recover from emit panic + expose watchdog tick (#1749) (#1810)
# Partial fix for #1749 — MQTT watchdog escalation + panic recovery +
tick exposure
Red commit:
|
||
|
|
b74a64ccfa |
fix(ui): canonical payload label map across packets/live/packet-filter (#1799) (#1804)
## Summary Replaces the three drifted per-surface payload-type label vocabularies with a single canonical map keyed by firmware enum name. Per the locked triage comment on #1799 ([comment-4823975431](https://github.com/Kpa-clawbot/CoreScope/issues/1799#issuecomment-4823975431)): > Create `public/payload-labels.js` exporting `{GRP_DATA: {short:'Group Data', long:'Group data packet', enumId:6}, ...}`. Migrate `packets.js typeMap`, `packet-filter.js FW_PAYLOAD_TYPES`, `live.js TYPE_COLORS legend` to consume it. E2E that scrapes each surface and asserts label equality. ## Changes - **`public/payload-labels.js`** (new) — canonical map exposed as `window.PayloadLabels` and `window.PayloadLabelsApi`. Keys are firmware enum names; values carry `{short, long, enumId}` plus derived `SHORT_BY_ID` / `FW_PAYLOAD_TYPES` / `TYPE_ALIASES` for legacy callers. - **`public/packets.js`** — `TYPE_NAMES` + `typeMap` now read from `PayloadLabelsApi.SHORT_BY_ID`. Literal kept only as a defensive fallback for the case where the script tag fails to load. - **`public/packet-filter.js`** — `FW_PAYLOAD_TYPES` + `TYPE_ALIASES` now sourced from `PayloadLabelsApi`. Literal fallback retained so `node test-packet-filter.js` still works headlessly. - **`public/live.js`** — legend `<li>` rows are now generated from `window.PayloadLabels` in stable order, killing the third-vocabulary `Message — Group text` / `Direct — Direct message` drift the #1797 review surfaced. - **`public/index.html`** — `<script src="payload-labels.js">` loaded before `roles.js` / `packet-filter.js` / `packets.js`. - **`test-issue-1799-label-vocab-e2e.js`** (new) — Playwright E2E. Scrapes `#liveLegend` rows and the `/packets` type-filter checklist, asserts each label matches `window.PayloadLabels[ENUM].short` for `TXT_MSG`, `GRP_TXT`, `GRP_DATA`. Also verifies `window.PacketFilter` still recognises the enum names. - **`.github/workflows/deploy.yml`** — wired the new E2E into the existing Playwright block. ## TDD trail - Red commit `eb392d4` — adds the failing E2E only (asserts `window.PayloadLabels` exists and labels match; both fail). - Green commit `44e902a` — introduces the canonical map and migrates the three surfaces. ## Verification - `node test-packet-filter.js` — 92/92 pass with the new fallback wiring. - Preflight: `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — clean. Browser verified: E2E `test-issue-1799-label-vocab-e2e.js` exercises `/live` legend + `/packets` type filter against a Playwright headless Chromium; CI's Playwright block runs it on every push. E2E assertion added: `test-issue-1799-label-vocab-e2e.js:139` — `assert(fromLegend === canon, ...)` and `assert(fromPackets === canon, ...)` per enum. Fixes #1799 --------- Co-authored-by: mc-bot <bot@corescope> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: clawbot <clawbot@kpa.com> Co-authored-by: clawbot <bot@clawbot.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
9ae547ed7b |
test: de-flake distance-202 and anchor-bias tests (deterministic timing) (#1808)
Two server tests flaked intermittently and reddened CI on unrelated (frontend) PRs that merged master: - TestDistanceConcurrentRequestsDuringBuildReturn202 asserted all 10 concurrent requests get 202 'during the build window', but the lazy distance build on the tiny test DB finishes almost instantly, so on a fast machine some requests raced past it and got 200 (~50% flake). Add a nil-by-default distanceBuildHook seam on PacketStore (zero overhead in prod) that the test uses to hold the build open until all requests have been served — making the window guarantee deterministic. - TestHandleNodePaths_AnchorBiasInconsistency_Issue1278 queried /paths right after store.Load(), racing the path-hop index that Load() builds in a background goroutine (#1008); the membership/canonical result was thus non-deterministic (rarer flake, worse under suite load). Wait for PathHopIndexReady() before querying. Both run 30x green and pass -race. No production behavior change (hook is nil). Co-authored-by: Waydroid Builder <claude@michael.arcan.de> |
||
|
Kpa-clawbotandopenclaw-bot
|
ec0ebeda2f |
fix(#1793): WebSocket CheckOrigin allowlist (block cross-origin scrapers) (#1795)
## Summary Closes the wide-open `/ws` WebSocket upgrader (`CheckOrigin: return true`) that lets any browser origin scrape live packet data. Replaces it with an explicit allowlist consulted from `cfg.CORSAllowedOrigins`, plus an implicit same-origin allowance and an empty-Origin (non-browser client) allowance. Fixes #1793. ## Rules (`Hub.checkOrigin`) - Empty `Origin` header → **allow** (non-browser clients; per-IP rate/deny gating tracked separately in #1794). - `Origin` host == request `Host` (case-insensitive) → **allow** (same-origin). - `Origin` matches an entry in `cfg.CORSAllowedOrigins` by exact case-insensitive match → **allow**. - `"*"` in `cfg.CORSAllowedOrigins` is **deliberately ignored** for `/ws`. A startup `[ws] WARNING:` is logged once when present. - Anything else → **reject** (gorilla returns 403). ### Deliberate divergence from CORS XHR CORS XHR (`corsMiddleware`) still honors `"*"` for read-only cross-origin GETs. The `/ws` upgrade does NOT, per OWASP's WebSocket Security Cheat Sheet: > Use an allowlist, not a denylist. Avoid wildcards or substring matching. — https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html `"*"` on the WS path would re-open the exact CSWSH/scraping vector this PR closes, so it is rejected with a startup warning rather than silently honored. This intentional asymmetry is documented in the updated `_comment_corsAllowedOrigins` in `config.example.json`. ## TDD red → green - `e5974c6a` **RED** — adds `cmd/server/websocket_checkorigin_test.go` with five cases; `SetAllowedOrigins` introduced as an enforcement stub so the test compiles and fails on the assertion (CI fails on this commit by design). - `a4791dc3` **GREEN** — implements `Hub.checkOrigin`, wires `SetAllowedOrigins` from `main.go`, updates the config example. All tests pass. ## Tests added (`cmd/server/websocket_checkorigin_test.go`) - `TestCheckOriginRejectsForeignOrigin` — foreign Origin → 403 - `TestCheckOriginAllowsEmptyOrigin` — non-browser client → 101 - `TestCheckOriginAllowsSameHost` — same-origin → 101 - `TestCheckOriginAllowsAllowlistedOrigin` — exact allowlist match → 101 - `TestCheckOriginWildcardDoesNotAllowForeignOrigin` — `"*"` in allowlist still rejects foreign origin → 403 ## Files changed - `cmd/server/websocket.go` — `Hub.allowedOrigins`, `SetAllowedOrigins`, `checkOrigin`, wired into `Upgrader.CheckOrigin`. - `cmd/server/main.go` — `hub.SetAllowedOrigins(cfg.CORSAllowedOrigins)` at the single call site. - `cmd/server/websocket_checkorigin_test.go` — new test file. - `config.example.json` — updated `_comment_corsAllowedOrigins` to document `/ws` gating and the `"*"` divergence. ## Out of scope (follow-up) - **#1794** — per-IP rate limit / deny list / connection cap for non-browser clients (which still bypass Origin because they don't send one). Layered defense; not in this PR. ## Verification - `go test ./cmd/server/...` — all server tests pass locally (574s). - Preflight clean (`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`). --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Michael J. Arcan
|
ae2e3933dd |
feat(server): store memory diagnostics + drop redundant obs.RawHex (#1773)
Drops the redundant per-observation RawHex (~98MB on a live store; reader already falls back to tx.RawHex #881) and adds an opt-in /api/perf?mem=1 memory breakdown (flood-forward share + per-component bytes). Profiled against a live instance. **Savings substantiation:** live-instance profiling shows ~1.66M observations in the store, each previously carrying its own per-observation `raw_hex` (avg ≈118 hex chars ≈59 bytes) that exactly duplicates the parent transmission's `raw_hex`. Dropping the duplicate on every load/ingest path eliminates ≈98 MB of redundant in-memory storage plus ~1.66M string allocations, with no data loss — the read path (`enrichObs`) already falls back to `tx.RawHex` when `obs.RawHex` is empty (verified by the new safety-gate test). The patched build cannot be run against the live instance here; instead the new opt-in `/api/perf?mem=1` diagnostic lets operators measure the real before/after (`trackedMB` and the per-component breakdown) directly after deploy. |
||
|
Kpa-clawbotandclawbot
|
b3189c613a |
fix(#1802): decode CONTROL DISCOVER_REQ/RESP subtype + body fields (#1806)
## Summary Extend CONTROL packet decoding to surface DISCOVER_REQ / DISCOVER_RESP subtype plus body fields in the packet detail view. Previously only the byte0 zero-hop flag was decoded; the body was rendered as opaque hex. ## What changed **Backend** — `cmd/ingestor/decoder.go` `decodeControl()` - New `Payload` fields (all omitempty): `CtrlSubtype`, `CtrlFilter`, `CtrlTag`, `CtrlSince`, `CtrlNodeType`, `CtrlSNR`, `CtrlPubKey`. - Subtype derived from `byte0 & 0xF0`: `0x80` → `DISCOVER_REQ`, `0x90` → `DISCOVER_RESP`, otherwise `UNKNOWN`. - REQ body parsed when `len(buf) >= 6`: `filter:u8 | tag:u32 LE`, plus optional `since:u32 LE` when 4 more bytes remain. - RESP body parsed when `len(buf) >= 6`: `node_type` (low nibble of byte0), `snr:i8`, `tag:u32 LE`, and `pubkey` hex — 32 bytes when full, 8 bytes when prefix-only. - Every field gated on length; short/truncated bodies emit subtype only and never panic. - `CtrlZeroHop` retained for backwards compatibility (rename flagged for follow-up per triage). **Frontend** — `public/packets.js` `getDetailPreview()` - New `decoded.type === 'CONTROL'` branch renders subtype + present body fields (filter / tag / since / node_type / snr / pubkey). Each field shown only when populated, so truncated CONTROL still gets a subtype label. ## Wire format reference - `firmware/src/Mesh.cpp:69` — `CTL_TYPE_NODE_DISCOVER_REQ=0x80`, `CTL_TYPE_NODE_DISCOVER_RESP=0x90`. - `firmware/examples/simple_repeater/MyMesh.cpp:773-820` — body parse / build. ## Tests (red → green, per AGENTS.md STRICT TDD) - `cmd/ingestor/issue1802_test.go` — 6 cases: REQ full body (with since), REQ no-since, RESP 32B pubkey, RESP 8B prefix pubkey, RESP truncated pubkey (no panic, no pubkey emitted), short body (subtype only), unknown subtype. Red commit `43713d3a` → green commit `d4b28180`. Pre-existing CONTROL tests (`TestDecodeControlZeroHop`, `TestDecodeControlMultiHop`) still pass. - `test-packets.js` — 3 cases on `getDetailPreview`: DISCOVER_REQ (filter+tag rendered), DISCOVER_RESP (snr+pubkey rendered), UNKNOWN subtype label. Red commit `be23e349` → green commit `845d6c48`. ## Preflight overrides - `check-branch-clean` (cross-stack): justified — issue #1802 explicitly spans backend decoder (`cmd/ingestor/decoder.go`) and frontend renderer (`public/packets.js`) per triage comment. Tests in both layers. Single-purpose PR. ## Scope discipline Files touched: `cmd/ingestor/decoder.go`, `cmd/ingestor/issue1802_test.go`, `public/packets.js`, `test-packets.js`. No other files. No firmware changes. No `cmd/server/decoder.go` changes. No `CtrlZeroHop` rename (deferred per triage). Fixes #1802 --------- Co-authored-by: clawbot <bot@meshcore.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
3efa37c46c |
feat(server): complete the #672 4-axis repeater usefulness score (#1762)
Adds Coverage (harmonic reach) + Redundancy (Tarjan articulation) axes + composite & grade. Closes #672. **TDD note (BLOCKER-1):** Community PR delivered as a single squashed commit, so there is no separate pre-fix failing-test commit — please accept as a community-PR exemption. The tests are *gating*, not just thorough: each axis test pins a specific topology outcome (coverage on line/star/disconnected/weight-sensitive; redundancy online/triangle/star/bridged-cliques), and an end-to-end `/api/nodes` surface test drives the whole pipeline and asserts the composite diverges from the Traffic axis. Inverting the `1/weight` distance, dropping the NaN/Inf reject, removing the `redundancyMinWeight` floor, or aliasing `usefulness_score` back onto `traffic_share_score` each break a specific assertion. The axis functions are pure (no hidden state), so the suite fully characterises the behavior without the red anchor. Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
|
17654dd090 |
docs(api): document per-node usefulness metrics in OpenAPI (#1769)
Documented Node schema (the four #672 usefulness axes + composite + A-F grade + relay fields) and response schemas on the node endpoints. Documentation-only; no behaviour change. Pairs with #1762 (documents the metrics it adds). Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Michael J. Arcan
|
fc26fb6b3a |
feat(#1751): show transported region scopes in repeater sidebar (#1752)
Closes #1751. |
||
|
|
f0763aecce |
fix(#1726): clear stale "varies" hash size once a node settles (#1788)
Fixes #1726. ## Problem A MeshCore v1.16.0 repeater configured for 2-byte path hashes (`path.hash.mode=1`) — e.g. `36f6c7c7…` (`DK_3400_RAK_TEST`) — kept showing as **"varies"** / mixed 1-byte + 2-byte for the full 7-day advert window. Per the live data in the issue triage: of the node's ~20 recent adverts, exactly **one** (2026-06-09, across 15 distinct observer paths) was a genuine 1-byte flood advert; every other advert was 2-byte. The flip-flop heuristic in `computeNodeHashSizeInfo` weighs that stale advert equally with recent ones, so an operator who flips `path.hash.mode` mid-flight (or a single old 1-byte advert) stays flagged for the full window with no way to signal "the config is settled now." ## Fix Two coupled changes in `cmd/server/store.go` `computeNodeHashSizeInfo`: 1. **Chronological ordering.** `byPayloadType[4]` iterates in insertion order, not timestamp order, so `HashSize = Seq[last]` could pick the wrong advert under out-of-order MQTT ingest or chunked cold-load (the "carmack" concern from triage). We now collect `(FirstSeen, size)` pairs and **stable-sort by `FirstSeen`**; ties keep insertion order, preserving prior behavior when timestamps are equal. 2. **Recency decay.** After `transitions >= 2` raises the flip-flop flag, clear it when the most recent `hashSizeRecentAgreeCount` (= **3**) non-zero-hop adverts all agree on a single size. A node still flapping (recent adverts disagree) stays flagged. `3` mirrors the existing ≥3-observation threshold used to raise the flag. ## Policy note Triage marked this **needs-operator-input** because the decay is a behavior/policy change. This PR implements the rule the triage proposed ("if the last 3 adverts agree, clear inconsistent"), which matches the reporter's stated expectation. Happy to adjust the threshold or gate it differently per your call. ## Tests `cmd/server/issue1726_hash_decay_test.go`: - `TestIssue1726_SettledNodeNotInconsistent` — reporter's case (`[2,1,2,2,2]` within window) → `Inconsistent=false`, `HashSize=2`. - `TestIssue1726_HashSizeUsesChronologicallyLatest` — out-of-order insertion still reports the chronologically-latest size. - `TestIssue1726_ActiveFlapperStaysInconsistent` — a node whose recent adverts disagree stays flagged. Existing flip-flop / hash-collision tests unchanged and green; full `cmd/server` package suite passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Erwin Fiten <e.fiten@opteco.be> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d437958474 |
fix(map): pin APC (Napa) and STS (Sonoma) observers (#1786) (#1787)
Fixes the map-coordinate gap in #1786. ## Problem Observers tagged with IATA code **APC** (Napa County) or **STS** (Charles M. Schulz–Sonoma County) render with no location and never pin on the map. ## Root cause `iataCoords` in `cmd/server/routes.go` is a hardcoded `IATA -> lat/lon` lookup used purely for placing observer/region markers on the map. It had no entry for APC or STS, so those observers had no coordinates to render with. This is **display-only**. Ingestion is not gated on these codes: `IsObserverIATAAllowed` (`cmd/ingestor/config.go`) short-circuits to `true` when the observer IATA whitelist is empty — which is the staging configuration. The reporter''s "packets disappear entirely" symptom is therefore **not** explained by this code path (likely an upstream `meshcoretomqtt`/broker topic issue; needs operator `mosquitto_sub` confirmation per triage). ## Fix - Add `APC {38.2132, -122.2807}` and `STS {38.509, -122.8128}` to `iataCoords`, matching the airports'' published coordinates. - Add a regression test (`TestIataCoordsIncludesNapaAndSonoma`) asserting both are present with the expected coordinates. ## Verification - `go test ./cmd/server/` — full package passes (`ok`). - `go vet ./cmd/server/` — clean. ## Scope note Checked the repo for other statically-enumerable region codes (`config.example.json` regions: SJC/SFO/OAK/MRY) — all already covered. The broader "are other in-use codes missing" question can only be answered against the live `cfg.Regions` + `db.GetDistinctIATAs()` set, which is operational, not in-tree. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Erwin Fiten <e.fiten@opteco.be> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
57956712e7 |
fix(#1768): Relay Airtime Share uses LoRa Time-on-Air (preamble-aware) — partial fix (#1776)
Partial fix for #1768 — Relay Airtime Share now uses closed-form LoRa Time-on-Air instead of a payload-bytes-only proxy, removing the ~3-4× bias against small frames (preamble + fixed-symbol intercept). cross-stack: justified — backend score formula needs a frontend caption change (`public/analytics.js` dumbbell preset banner + tooltip) so operators can interpret the assumed PHY block. Both move together or the metric is misleading. ## Red commit `8da57062` — failing test asserts ToA-based score (~83.48 % ADVERT share on the locked acceptance fixture) instead of the byte proxy's 95.24 %. `internal/lora.TimeOnAir` was a zero-returning stub at the red commit; tests failed with assertion errors, not build errors. ## Green commit `dd402edd` — implements `lora.TimeOnAir` (Semtech AN1200.13 / SX126x §6.1.4 closed form, cross-checked against RadioLib), wires `score = TimeOnAir(payloadBytes, preset) × distinctRelays` in `cmd/server/relay_airtime_share.go`, surfaces the preset in the JSON response and analytics caption. ## Config (per AGENTS Config Documentation Rule) New keys under existing `analytics` block: ```json "loraPreset": { "freq": 869600000, "bw": 62.5, "sf": 8, "cr": 5 } ``` Defaults match the deployment's actual `get radio` (869.6 MHz / BW 62.5 kHz / SF 8 / CR 4/5). `CRC=1`, `IH=0`, `DE = (T_sym ≥ 16 ms)`, and the SF-dependent preamble (32 for SF≤8 else 16, per firmware `preambleLengthForSF` / MeshCore PR #1954) are firmware-fixed constants in `internal/lora/toa.go` and intentionally NOT surfaced as config (per re-triage). ## Scope In-scope files (6): - `internal/lora/toa.go` (new package — closed-form ToA) - `internal/lora/toa_test.go` (table-driven preset tests) - `cmd/server/relay_airtime_share.go` (wire ToA into score) - `cmd/server/relay_airtime_share_test.go` (recomputed expected values) - `cmd/server/config.go` + `config.example.json` (preset config keys) - `public/analytics.js` (preset caption on dumbbell chart + tooltip) Plus `cmd/server/go.mod` (replace directive for the new internal module). ## Deferred to v2 (separate issues per re-triage) - Per-observation SF/BW + radio-settings-aware dedup (blocked: ingestor stores SNR/RSSI only, no SF/BW on observations). - CR-per-hop dual-point sensitivity band (CR scales only the payload symbol term `(CR+4)`, not the preamble/header; second-order accuracy gain). - Cross-SF bridge accounting. ## Tests ``` cd internal/lora && go test ./... → PASS cd cmd/server && go test -run RelayAirtime → PASS ``` ## Preflight overrides - `check-branch-clean` (cross-stack): justified above — score formula change requires matching caption update; both files trace to the same issue. --------- Co-authored-by: kpa-clawbot <kpa-clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@openclaw.local> Co-authored-by: bot <bot@meshcore> |
||
|
Kpa-clawbotandclawbot
|
e465e1c6c6 |
perf(#1740): replace idx_tx_last_seen with partial index WHERE last_seen=0 (#1756)
Fixes #1740. ## What Replace the full `idx_tx_last_seen` with a partial index `WHERE last_seen=0`. Two ordered, sequential migrations in `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn`: - **(a)** `CREATE INDEX IF NOT EXISTS idx_tx_last_seen_zero ON transmissions(id) WHERE last_seen=0` - **(b)** `DROP INDEX IF EXISTS idx_tx_last_seen` — gated after (a) succeeds (sequential `Exec`; (b) never runs if (a) errors) ## Why The only consumer is `chunkedTxLastSeenBackfill`'s `WHERE last_seen=0` scan + `MAX(id)` lookup. The full index covers ALL rows including the long tail where `last_seen != 0` after backfill converges (71K+ rows in prod per carmack's #1740 note). The partial index degenerates to ~the count of un-backfilled rows (0 in steady state, bounded by ingest rate during ops) and stops competing for page cache. ## Migration cost Both migrations are sync and annotated `PREFLIGHT: async=false`: - (a) `CREATE INDEX` on partial subset `WHERE last_seen=0` is bounded by un-backfilled rows — a small superset of the inflight ingest window, not a full table scan. - (b) `DROP INDEX` is a metadata-only schema rewrite in SQLite — no row scan at any size. `dbschema/` has no access to `Store.RunAsyncMigration` (that helper lives in `cmd/ingestor/` and is the wrong layer for the schema source-of-truth per #1321), so sync is the only path here regardless. ## TDD - **RED** `a529f0f4`: `EXPLAIN QUERY PLAN` test asserting the backfill `MAX(id) WHERE last_seen=0` query uses `idx_tx_last_seen_zero` + a second test asserting the legacy `idx_tx_last_seen` is dropped post-Apply. Both failed on assertion (planner picked the full index; partial index didn't exist). - **GREEN** `208fde8a`: add migrations (a) and (b). Both tests pass. ## Files touched - `internal/dbschema/dbschema.go` — swap the index - `internal/dbschema/dbschema_test.go` — `EXPLAIN QUERY PLAN` pin + DROP assertion - `cmd/ingestor/db.go` — comment refresh only (idx_tx_last_seen → idx_tx_last_seen_zero) ## Acceptance - ✅ New partial index created - ✅ Old full index dropped via gated migration (sequential, order preserved) - ✅ Query plan test asserts partial-index usage - ✅ Existing migration tests still green (`internal/dbschema`, `cmd/ingestor` TestIssue1690 + applySchema) --------- Co-authored-by: clawbot <bot@openclaw> |
||
|
|
22fe929da2 |
feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728)
Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be> |
||
|
Kpa-clawbot
|
97833c523b |
fix(post-packets): use v3 observations schema (closes #1196) (#1704)
## Summary `POST /api/packets` is broken on every v3-schema install — which is the default since #1289. The handler issues two writes against legacy v2 column names and silently swallows the observation insert's error, returning `200 OK` with `id>0` while persisting zero observation rows. ## Root cause `cmd/server/routes.go:1225-1235` (pre-fix) used the v2 schema shape: ```go INSERT INTO transmissions (... path_json ...) // path_json removed in v3 INSERT INTO observations (transmission_id, observer_id, observer_name, snr, rssi, timestamp) // v2 columns // timestamp written as RFC3339 text; v3 wants unix INTEGER // second Exec's error was discarded ``` v3 schema (`cmd/ingestor/db.go:289-304`): `observations.observer_idx INTEGER` (FK `observers.rowid`), `observations.timestamp INTEGER` (unix epoch), `path_json` lives here not on `transmissions`. Reporter [@EldoonNemar](https://github.com/EldoonNemar) called this out precisely in #1196 — both the schema mismatch and the divergence between the test harness (which uses the v3 shape) and the handler (v2 shape). ## Fix `cmd/server/routes.go`: - `transmissions` insert: drop `path_json` column. - Observer resolution: `INSERT OR IGNORE INTO observers (id, name, ...)` then `SELECT rowid` — mirrors the ingestor resolver at `cmd/ingestor/db.go:778,906`. - `observations` insert: write `observer_idx INTEGER` + `timestamp = time.Now().Unix()`; `path_json` moved here. - **Propagate both insert errors** (transmission + observation) as `500` instead of swallowing them. ## TDD | Step | Commit | Result | | ----- | ------- | ------ | | RED | `46d25389` | Test fails on master: `id=0` because the transmissions insert references a column not present in v3. | | GREEN | `dae57d67` | Test passes; round-trip persists the observation with `observer_idx` resolved from the seeded `obs1` row and a unix-epoch `timestamp`. | Local repro: ``` # RED on the test commit alone: $ go test -run TestPostPacketPersistsV3Schema -count=1 . --- FAIL: TestPostPacketPersistsV3Schema (0.03s) routes_test.go:4755: expected transmission id > 0, got 0 (body: {"id":0,"decoded":{...}}) FAIL # GREEN on HEAD: $ go test -run TestPostPacketPersistsV3Schema -count=1 . ok github.com/corescope/server 0.037s ``` ## Scope Two files, both in `cmd/server/`: - `cmd/server/routes.go` (+38/-12) — handler rewrite - `cmd/server/routes_test.go` (+66) — round-trip regression test No public API signature changes. No DB schema changes (consumes the existing v3 schema correctly). Closes #1196 |