Matches the term used everywhere else in CoreScope (/api/observers,
observer names, etc.) instead of an inconsistent synonym.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetChannelMessages previously used "first observation wins" for a ping's
botReply -- if the same flood was heard by multiple stations, only
whichever one happened to be scanned first shaped the hop count, SNR, and
relay path. That understates reach: different stations legitimately hear
the same flood at different hop depths depending on which relay leg
reached them.
Now tracks, across every observation of the ping: the DEEPEST (max-hop)
observation's hops/SNR/relay path, and every DISTINCT station that heard
it. The reply shows the deepest leg's path and reports "heard by N
stations" once more than one did (falling back to the single station's
name when there's exactly one, same as before).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
REST-loaded channel history (GetChannelMessages) now resolves the ping
message's relay path to node names ("via RepeaterA → RepeaterB"), bulk-
resolving every referenced pubkey across the page in one query rather
than per-message. Falls back to the raw pubkey when a hop's node isn't
known. Also includes the message's region scope and (via a handler-level
pass, since area resolution needs server config db.go doesn't have)
its resolved area.
The client-side pingBotReply (WebSocket live-push + PSK-channel decrypt
paths) gains scope/area too, since both are already present in that data.
Relay-path names are REST-only for now: the live WS broadcast doesn't
carry a resolved_path, only historical/REST-loaded messages do.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A channel message whose text is exactly "ping" (mention-prefix like
"@MeshviewBot ping" stripped first) now gets a synthesized "pong" reply
showing hop count, SNR, and hearing observer -- computed at read time
from that message's own already-stored data, no new table.
Deliberately NOT transmitted back onto the mesh: CoreScope has no publish
path to a MeshCore broker/radio (confirmed: it only ever subscribes to
MQTT, never publishes). The reply is visible only in CoreScope's own
Channels view, rendered as a visually distinct dashed-border bubble with
an explicit "Not sent to the mesh" caveat so it's never mistaken for a
real bot reply the sender's own radio received.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MeshCore firmware sends two capability bytes on ADVERT packets (wire bits
per AdvertDataHelpers.h) whenever the HasFeat1/HasFeat2 flags are set.
CoreScope already decoded them per-packet but discarded them rather than
storing a per-node value.
- New feat1/feat2 columns on nodes/inactive_nodes (internal/dbschema,
idempotent ALTER like the existing multibyte_sup/multibyte_evidence
columns).
- Ingestor's UpdateNodeTelemetry now writes feat1/feat2 alongside
battery_mv/temperature_c in the same COALESCE-based UPDATE, from the
same ADVERT payload.
- Server exposes them on /api/nodes and /api/nodes/{pubkey} (nullable,
same shape as battery_mv/temperature_c).
- Node detail page shows them as raw hex (0x....) in the Overview panel
when present -- undecoded, since CoreScope doesn't know the individual
bit meanings, but no longer silently discarded.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetHopDepthAnalytics now buckets its scoped/unscoped hop-index tallies by
time (same 5min/1h/6h bucketing as GetScopeStats' TimeSeries) and computes
a per-bucket median for each series, exposed as a new TimeSeries field.
Median is a *int, nil when a bucket had no traffic of that kind -- 0 is a
valid median hop, so absence has to stay distinguishable from zero.
Frontend renders it as a two-line SVG trend chart on the Scopes > Hop
Depth sub-tab (same visual language as Overview's scoped/unscoped
time-series), answering "is containment getting better or worse over the
window" rather than just a single window-wide snapshot. Each series'
polyline is built from contiguous non-null segments so a gap renders as
a visible break instead of silently interpolating through missing data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the #1812 per-node relay hop-count work with a network-wide view:
GET /api/analytics/hop-depth answers (1) whether scoped traffic actually
travels fewer hops than unscoped before hitting a repeater's flood.max cap
(Scopes tab Overview: new "Flood Containment" comparison), and (2) which
repeaters relay unscoped traffic that already traveled far vs merely
locally (Foreign Traffic tab: min/median/max hop columns joined onto the
existing unscoped-relay table by public key).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AreaScopeAdoption.Matching entries now carry MatchedScopes (which of
the area's linked regionScopes each node actually matched, via
default_scope or by relaying it) instead of just a bare name/key ref.
A node can match more than one when an area links several scopes and
the node uses/relays more than one of them.
Frontend: an area linking more than one scope (e.g. Europa's "eu" and
"europe") now splits its Supporting list into one sub-group per scope
instead of a single flat list, so it's visible which nodes support
which specific scope.
AreaEntry.RegionScope (single string) -> RegionScopes ([]string), so a
broad umbrella area (e.g. Europa) can link more than one scope name
(e.g. both "eu" and "europe") -- a node matching any one of them now
counts as supporting the area in computeScopeAdoptionByArea, the
"Scope Adoption by Area" section, and the regionScope->label lookup
used to annotate region codes elsewhere on the Scopes tab.
dborup: sitting in Aarhus but sending with the broad #dk scope should
still show "Aarhus by" -- the scope-linked area alone doesn't tell you
where the sender physically was. Adds a second, independent area
resolved from the message's own path[0] entry-point repeater (same
unique_prefix-only discipline as resolveEntryPointArea/Wardriving),
shown as "From: <area>" alongside the existing "Scope: #dk (Danmark
alle)" tag.
GetChannelMessages (both DB and in-memory paths) now captures path[0]
as an internal "entryPrefix" field; handleChannelMessages resolves it
to an area server-side via the existing resolveEntryPointArea, then
strips entryPrefix before the response goes out -- raw hash prefixes
never reach the client for this feature.
Scoped to the REST message list only, not the WebSocket live-append
path (shared broadcast infra used by several other pages) -- a
brand-new live message shows the scope-linked area only, until the
next full load picks up the resolved path[0] area.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: "Danmark (alle)" showed almost nothing (0 of 10 support) even
though most real nodes use the generic #dk scope. Root cause: the
per-area node count used AreaKeyForPoint, which picks only the single
most-specific area for each node -- a node in "Odense by" never also
counted toward the broader "Fyn" or "Danmark (alle)" it geographically
sits inside, so a country-level area only ever saw the leftovers no
smaller area had already claimed.
New AreaKeysForPoint returns every containing area (not just the
best match), used by computeScopeAdoptionByArea so a node now counts
toward all of its containing areas -- Danmark (alle) genuinely
aggregates every Danish sub-area's nodes now, not just stragglers.
AreaForPoint/AreaKeyForPoint (single-match, used by the GPS-share and
session area badges) are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup wanted per-node visibility, not just aggregate counts: for
each area with a linked region, list the actual nodes that support
it (own default_scope or ever relayed it) vs. the ones that sit there
but don't. Replaces the summary table with expandable per-area groups
(same pattern as Repeaters by Region), each showing a Supporting /
Not Supporting node list with links.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: a repeater that has relayed dk-horsens traffic supports the
Horsens area, even if its own default_scope is something else (e.g.
the generic #dk most nodes actually use) or unset entirely. The
previous version only checked default_scope, missing this -- same
runs-this-region vs carried-this-region's-traffic distinction the tab
already draws between OriginatingNodesByRegion and RepeatersByRegion.
Both NodesWithAnyScope and NodesMatchingArea now also check the
node's entry in the cached RepeaterRelayInfo map (TransportedScopes),
reusing the same cache RepeatersByRegion already populates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New geographic view, independent of the raw hashRegion-code-based
Region Utilization: buckets every positioned node by its configured
area (AreaKeyForPoint) and tallies how many have any default_scope
at all, and how many specifically match the area's own linked
region. Surfaces gaps Region Utilization can't see, since that only
knows about region strings that already appeared in traffic — a real
area with real nodes that never produced a single scoped message is
invisible there but shows up here as 0% adoption.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For senders who never share a literal GPS fix, resolve the session's
path[0] entry-point prefix to a known repeater position (only when it
resolves unambiguously, same unique_prefix discipline as
/api/resolve-hops) and label it with the most specific configured
area — shown as a badge in the Sessions table, clearly marked
approximate since it's the repeater's position, not the sender's own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds AirtimeMs to WardrivingSession: total network airtime consumed
relaying a session's messages, using the exact same formula as the
Overview tab's "Relay Airtime Share" (issue #1768) — ToA(payload_bytes)
x COUNT(DISTINCT resolved repeater in path), summed per message.
That formula needs the in-memory store's resolved-path index
(resolvedPubkeyReverse), which db.go's SQL-only wardriving code
doesn't have — so buildWardrivingSessions now also tracks each
session's transmission IDs (unexported, json:"-"), and the route
handler calls the new PacketStore.AirtimeForTransmissions(txIDs) to
fill in AirtimeMs after the fact. Omitted entirely (nil) in DB-only
mode rather than shown as a misleading zero.
Frontend adds an Airtime column to the Sessions table (ms/s
formatting, dash when unavailable).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found in production: at least one wardriving client appends plaintext
"<lat>,<lon>" after the standard token (e.g.
"MM:c3e_zJ1rUA:55.59743,13.00128"), confirmed against live traffic.
The generic anomaly framing (flagging non-standard payload length,
showing a raw hex dump) was based on the wrong assumption that this
was an undocumented binary format — it's actually deliberate,
human-readable coordinate sharing by that sender's client.
Replaces WardrivingAnomaly/detectWardrivingAnomalies/
StandardPayloadCount with WardrivingGPSShare/detectWardrivingGPSShares:
detects the specific "<token>:<lat>,<lon>" suffix (with range
validation) and reports the most recent position per sender. The
per-message drill-down gets Lat/Lon instead of a standard/anomaly
flag. Frontend replaces the "Payload Anomalies" section with "GPS
Sharing", linking each position to the live map
(#/map?lat=..&lon=..&zoom=15, an existing deep-link format).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/analytics/wardriving/sender-messages returns one sender's
individual messages (most-recent-first, capped at 200): resolved
entry-point path (path[0] first), per-observer SNR/RSSI, and payload
standard/anomaly classification. Pass since+until to scope to one
session's exact range; otherwise it covers the sender's whole window.
Frontend makes sender names in Top Senders and Sessions clickable —
clicking toggles an inline expansion row with that sender's messages,
resolving every distinct path prefix in one /resolve-hops call (same
unique_prefix-only discipline as the aggregate Entry Points table).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
decodeGrpTxt (cmd/ingestor/decoder.go) builds decoded_json.text as
"<sender>: <message>", so a wardriving message's real text is
"<sender>: MM:<base64>", not a bare "MM:<base64>" at the start of the
string. detectWardrivingAnomalies's HasPrefix(text, "MM:") check never
matched anything live, silently returning standardPayloadCount=0 and
no anomalies — caught after deploying idea 6 to stg and seeing 0/0
against traffic that should have had ~160 standard-format messages.
Fixed to match the exact "<sender>: MM:" prefix. The Go test fixtures
shared the same wrong assumption and are fixed too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with a payload-anomaly detector:
every "MM:<base64>" message is checked against the standard 7-byte
anonymous session token (confirmed empirically against live traffic).
Non-standard lengths or undecodable payloads are grouped per sender
as a candidate signal that MeshMapper's optional "Broadcast My
Coordinates" mode is active — surfaced as a raw hex dump, never
interpreted as lat/lon, since that mode's byte format is undocumented.
Messages without the "MM:" prefix (plain channel chat) are ignored
entirely rather than polluting either bucket.
Frontend adds a Payload Anomalies table plus a stat card, completing
all 6 originally-proposed wardriving analytics ideas.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with each sender's messages
grouped into distinct sessions/runs — a gap over 15 minutes starts a
new one. Each session reports duration, message count, and how many
distinct entry-point repeaters/observers it touched. Frontend adds a
Sessions table (most-recent-first) and a session-count stat card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with avgSnr/avgRssi over the
same time buckets as the activity series, plus overall averages.
Frontend adds two min/max-scaled line charts (SNR has no natural
zero floor, RSSI is negative dBm, so these can't reuse the 0-baseline
message-volume chart) and two stat cards.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/analytics/wardriving endpoint plus an Analytics tab
covering the three requested angles: activity over time + top
senders, entry-point repeaters (path[0] tally, unique_prefix-only
name resolution), and per-observer coverage using observers' known
IATA coordinates. MeshMapper's on-air ping is an anonymous session
token by default, not the sender's live GPS, so sender position
itself isn't tracked — documented in the tab and OpenAPI description.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scoped/Unscoped/Unknown counts answered "how much" but not "which
regions" — a channel spanning multiple hashRegions areas looked
identical to one using just one. New "Regions Used" column lists the
distinct scope_name values seen on each channel's scoped messages,
most-used first.
Backend: GetChannelScopeAdoption now also queries per-(channel,
scope_name) counts and attaches them as ChannelScopeAdoption.Regions.
Non-fatal on query failure — adoption counts still render without it.
Bot review on PR #1852 (munger, MAJOR-adjacent→MINOR): "bounded by the
1-byte channel hash space" only holds for encrypted channels (256
enc_%02x buckets) — plain-text CHAN channels use a free-form name
string (ingestor/db.go), so the bound isn't as hard as the comment
claimed. Correctness of the uncap itself is unaffected; this is a
doc-only fix.
GetChannelScopeAdoption was capped at the top 30 channels by message
volume, silently hiding quieter channels from the Scopes tab. Channel
cardinality is bounded by the 1-byte hash space, so the cap wasn't
protecting against anything real — removed it.
Client gets a filter (All/Unencrypted/Encrypted) based on the 'enc_'
channel_hash prefix db.go already uses for channels the ingestor
couldn't decrypt, so a visitor can separate plain channels from
encrypted ones without hunting through a long unfiltered list.
Working through the outstanding non-blocker findings across all five
automated review passes on the PR:
- MAJOR: GetChannelMessages' rows.Scan() error was silently discarded
— a schema mismatch would produce zero-valued messages instead of a
visible error. Now returns the error, matching sibling query loops.
- Extracted the hashRegions name-normalization rule (trim, "#"-prefix,
dedupe) shared between cmd/ingestor's loadRegionKeys and cmd/server's
region-utilization diff into a new internal/regions package, so the
two can no longer drift apart on the rule independently.
- Batched GetRepeaterNamesByKeys' SQL IN (...) clause in chunks of 500
— an unbounded clause risks SQLITE_MAX_VARIABLE_NUMBER on very large
deployments' byPathHop candidate sets.
- handleScopeStats' remaining silently-swallowed err==nil branches
(GetMatchedRegionNames, GetNodesByDefaultScope,
GetChannelMessageScopeStats, GetChannelScopeAdoption) now log a WARN
breadcrumb on failure instead of failing invisibly.
- Scope Adoption table was rendering 3 of the 4 ChannelScopeAdoption
fields (Unscoped omitted) — the visible numbers didn't reconcile to
the message total without doing the subtraction by hand. Added the
column.
- Removed a duplicate `vertical-align: middle` declaration on
.badge-transport (dead, not a behavior change).
- Deleted ChannelMessageResp — flagged for interface{} vs *string/*int
typing, but turned out to be completely unused dead code; removing
it resolves the finding more directly than retyping something
nothing constructs.
Left two NIT/MINOR items as-is with reasoning:
- renderRegionNodeGroups' inline styles match the same pattern used in
15+ other places in analytics.js — "fixing" only this one function
would make it less consistent with the file, not more.
- TransmissionResp's interface{} fields (a different struct than the
one just removed) are an established, actively-used pattern for
nullable SQL-scanned values across that whole response type;
retyping it is a much larger, unrelated refactor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds channelScopeAdoption to /api/scope-stats: the existing
ChannelMessages aggregate (scoped/unscoped/unknown for channel chat)
broken down PER CHANNEL — which specific channels (#test,
#wardriving, ...) actually use region scoping vs which never do.
Ordered by message volume, capped at the top 30 channels.
Frontend renders a compact table under "Channel Messages": channel
name, total messages, scoped count+%, unknown count.
Adds hourlyActivityByRegion to /api/scope-stats: each region's message
counts bucketed by hour-of-day (0-23 UTC), aggregated across every day
in the window — answers "when during a typical day is this region
active" rather than "how did volume change over the window" (that's
the existing chronological TimeSeries chart).
Frontend renders a compact heatmap: one row per region, 24 hour
columns, color intensity normalized per-row (each region's own busiest
hour) so a quiet region's daily shape stays visible next to a loud
one instead of being crushed toward zero.
Adds channelMessages to /api/scope-stats: the same scoped/unscoped/
unknown question as the main Summary, but restricted to payload_type=5
(channel chat) instead of all observed traffic. Most channel chat is
plain FLOOD rather than transport-scoped, so this can read very
differently from the all-traffic numbers — answers "how many of our
actual channel messages carry a region scope" directly instead of
requiring the reader to infer it from the broader stats.
New GetChannelMessageScopeStats() mirrors GetScopeStats' query shape
but scopes TotalMessages to ALL route types for payload_type=5 (not
just route_type 0/3), since restricting to transport routes would
answer a different question than "how many channel messages, period".
Frontend renders a small "Channel Messages" stat-card row under the
main summary cards, window-scoped like the rest of the tab.
Adds originatingNodesByRegion to /api/scope-stats: nodes whose OWN
default_scope (#899) is a given region, complementing the existing
repeatersByRegion (transported_scopes) breakdown. The distinction
matters — a repeater can relay traffic for a region it isn't itself
configured with, so "who runs this region" and "who has carried this
region's traffic" are different, both useful questions.
Frontend: refactored the per-region collapsible-list rendering (used
by both breakdowns) into a shared renderRegionNodeGroups() helper
instead of duplicating the HTML-building logic, and added a "Nodes
Running This Region" section alongside "Repeaters by Region".
byPathHop indexes both full pubkeys and short hex-prefix "bucket" keys
used internally for ambiguous-hop resolution — GetRepeaterRelayInfoMap
returns TransportedScopes for every one of those keys indiscriminately.
The first deploy of repeatersByRegion iterated the raw map and fell
back to showing the bucket key itself when no name matched, so a
handful of short internal keys were counted as "repeaters" (stg showed
594 "repeaters" for #dk — every active repeater plus every 2-6 char
bucket key that ever touched a #dk packet).
GetNodeNamesByKeys is now GetRepeaterNamesByKeys and filters
`role IN ('repeater','room')` in the SQL itself, so a key only survives
if it's a real node — bucket keys never match a nodes.public_key row
and are dropped rather than falling back to the raw key.
Adds repeatersByRegion to /api/scope-stats: for every region that has
ever matched a transmission, which distinct repeaters/rooms have
relayed traffic carrying that scope. Sourced from the same 5-min
background-recomputed bulk relay-info cache the Nodes page already
uses (GetRepeaterRelayInfoMap / TransportedScopes, #1751) — no new
expensive computation, just an inversion + name lookup.
Frontend renders a collapsible per-region repeater list (name links
to the node detail page) under a new "Repeaters by Region" section,
explicitly framed as a coverage/redundancy signal: a region carried
by only one repeater is a single point of failure for that area.
Adds configuredRegions/unusedRegions to /api/scope-stats: an all-time
(not window-scoped) diff between the operator's configured hashRegions
list and the set of scope_name values that have actually matched a
transmission still in retention. Surfaces how much of the region list
is dead weight — directly actionable evidence for pruning, which is
also the real fix for the HMAC-collision noise (fewer configured
regions -> lower birthday-collision probability per packet).
Server config now parses hashRegions (previously ingestor-only, same
config.json key) purely to read the configured names — no HMAC key
derivation happens server-side.
Frontend: a "Region Utilization" section on the Scopes tab shows
used/unused counts and a collapsible list of the unused region names.
The packet detail pane already showed scope (with an "unknown scope"
fallback for empty scope_name), but the packets table itself gave no
at-a-glance signal — the existing transportBadge() "T" marker only
encoded route type.
transportBadge() now takes an optional scopeName argument: a resolved
region enriches the tooltip, an empty scope_name (transport-eligible
but unmatched/ambiguous) renders as "T?" with a distinct muted badge
style instead of the confident amber, so it's visually distinguishable
from a resolved scope without relying on color alone. Existing callers
that don't pass scopeName (live.js) are unaffected.
QueryGroupedPackets (SQLite + in-memory) didn't select scope_name at
all — added it, since the Packets tab defaults to the grouped view.
GetChannelMessages already returned an empty scope string for
transport-eligible packets whose region couldn't be determined (no
configured region matched, or matchScope now reports an HMAC
collision as unknown), but the UI treated empty scope the same as
"not applicable" and rendered nothing — indistinguishable from a
plain FLOOD/DIRECT message that never carries a scope at all.
Adds route_type to the channel-message payload (both SQLite and
in-memory paths, plus the decrypt-candidate and live WS paths) so the
frontend can tell "not transport-scoped" (routeType 1/2, no tag) apart
from "transport-scoped but unresolved" (routeType 0/3 with empty
scope, now shown as "Scope: unknown").
Adds scope_name to GetChannelMessages (both SQLite and in-memory
store paths) and to the general packet response shape, then renders
it as "Scope: <name>" in the channel chat meta line so operators can
see which region scope a message was transported under.
Fixes#1838
## Problem
`/api/scope-stats` reported 100% scoped whenever any region was
configured. Reporter noticed on a scopeless instance that "unscoped" was
always zero — the pie visual is misleading to operators deciding on
`denyf *`.
## Root cause
`cmd/server/db.go:22` restricted the entire scope-stats denominator to
`route_type IN (0, 3)`. Per firmware `docs/packet_format.md § Route
Types`:
- `0` = `TRANSPORT_FLOOD`
- `1` = `FLOOD`
- `2` = `DIRECT`
- `3` = `TRANSPORT_DIRECT`
Only routes 0 and 3 carry `transport_code_1` (transport-level scope).
Routes 1 and 2 are inherently unscoped by protocol. The existing SQL was
correct for the "how many transport-scopable routes are actually scoped"
question, but the denominator was silently promoted to "all traffic" in
the UI. Bonus: the comment on `routeTypeTransportSQL` labelled routes
0+3 as "FLOOD (0) and DIRECT (3)" — wrong on both counts.
## Fix
- `cmd/server/db.go` — corrected the `routeTypeTransportSQL` comment;
added `routeTypeNonTransportSQL = "route_type IN (1, 2)"` alongside it.
- `GetScopeStats` runs a second `COUNT(*)` over `route_type IN (1,2) AND
first_seen >= ?` and folds that count into `Summary.Unscoped`. Same
index path as the existing query — one extra scan per `/api/scope-stats`
call (cached 30s per triage's carmack finding).
- `public/analytics.js` — Scopes tab header explains the denominator
(all observed transmissions) and which route types carry scope. Card
notes now render `X% of all traffic` for Scoped/Unscoped and `X% of
scoped` for Unknown Scope so the pie's denominator is explicit.
## TDD
- Red: `5554ffe4` — extended `TestGetScopeStats` +
`TestHandleScopeStats` with `route_type=1` and `route_type=2` rows and
asserted `Unscoped = 3` (1 transport-NULL + 2 non-transport). Ran the
tests and confirmed assertion failure (`Unscoped = 1, want 3`).
- Green: `ebbb9253` — implementation + label copy. Full `go test
./cmd/server/...` passes (54s).
## Preflight overrides
- check-branch-clean: justified — cross-stack fix by design (backend
semantics change + matching frontend label copy). All 4 files are
exactly the surface the triage comment identified.
## Verification
- `go test ./cmd/server/...` — 54s, all pass.
- Firmware confirmation: `firmware/docs/packet_format.md:20-24` (route
type table).
## Files touched
- `cmd/server/db.go` — comment fix + second COUNT query.
- `cmd/server/db_test.go` — extended fixture.
- `cmd/server/routes_test.go` — extended fixture + isolate from seed
data.
- `public/analytics.js` — labels and header copy.
---------
Co-authored-by: corescope-bot <bot@corescope.dev>
## #1690 — cold-load uses wrong time axis (RED → GREEN)
The on-disk DB has thousands of long-lived hashes with recent traffic.
Prod's
cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a
column
that is set once at insert time and never updated — so re-observation of
an
old hash does not move it into the hot window. Result: prod cold-loaded
~0.3%
of the on-disk rows and flipped `backgroundLoadComplete=true` without
ever
walking the retention window (the `retentionHours - hotStartupHours <=
0`
short-circuit at line 1353 of `cmd/server/store.go`).
### Three sub-fixes
**A) Denormalize `transmissions.last_seen`** so cold-load can window on
effective recency.
- `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn`
adds the
column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both
PREFLIGHT-annotated as cheap metadata-only ops).
- `cmd/ingestor/db.go::OpenStoreWithInterval` schedules
`tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` —
`UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE
last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod).
- Writer-side: `InsertTransmission` seeds `last_seen` on initial insert,
and every observation insert bumps `last_seen = ?` via prepared
statement
`stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest
never goes backwards).
- Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and
`cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY
clauses to `t.last_seen` when the column is present (PRAGMA-detected via
`DB.hasLastSeen`). Test/legacy DBs without the column fall back to
`first_seen` so existing fixtures stay green.
**B) Honest `backgroundLoadComplete` gating.**
- Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod
runs
with both at 12h, which flipped Done=true immediately.
- After the chunk loop, query
`SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor`
and
compute `loadCoverageRatio = inMem / inDB`. Done=true only when
`ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` +
`backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000
rows (1000 in memory)"`).
- `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the
perf
endpoint can read them without blocking the writer.
**C) Perf exposure.**
`PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`,
`LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of
the
on-disk DB the in-memory store currently reflects, so operators can see
the
0.3% case in `/api/perf` without reading the logs.
### TDD trail
- **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests +
21-line
store.go scaffolding. CI on this commit failed on assertions (intended).
- **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite:
`Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`,
`Test1690_PerfStats_NewFields`, `TestHotStartup_*`,
`TestIssue1690_LastSeenUpdatedOnObservation` — all pass.
Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()`
guard around `backgroundLoadDone.Store(true)` —
`Test1690_BackgroundLoadHonesty`
fails on the assertion `"backgroundLoadDone=true with only 1000/5000
packets
loaded; must be false until coverage ≥ 90%"`. Restored.
### Async-migration preflight
- `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both
`// PREFLIGHT: async=true reason="..."` annotated.
- `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`.
- `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level
UPDATE BY PRIMARY KEY, not a migration.
### Preflight overrides
PREFLIGHT-MIGRATION-SCALE: <30s N=5K
- check-async-migration: justified for
`cmd/server/issue1690_cold_load_test.go`
CREATE TABLE/INDEX statements — these build an in-memory test fixture DB
(≤5000 rows, runs in <1s in CI), not a prod migration.
Fixes#1690.
---------
Co-authored-by: meshcore-bot <bot@meshcore.local>
Co-authored-by: bot <bot@example.com>
Closes#1290.
cross-stack: justified — backend persists firmware-side `repeat` hint to
a new observers column, frontend surfaces the listener/repeater status
as a badge on the observers list and node-detail Heard By table per the
issue's UI acceptance criterion.
## What
Firmware 1.16 publishes a `repeat: on|off` flag in the MQTT `/status`
JSON (confirmed by @cwichura on the issue thread — see
[`MQTTMessageBuilder.cpp:58`](https://github.com/agessaman/MeshCore/blob/b45373a31f111fb0de98bb3b168226d09ceadc47/src/helpers/MQTTMessageBuilder.cpp#L58)
in `agessaman/MeshCore mqtt-bridge-implementation-flex`). Listener-only
observers (`repeat:off`) by firmware contract never relay packets, so
they cannot legitimately be a hop in someone else's resolved path. This
PR plumbs the hint end-to-end so the disambiguator stops considering
them.
## How
* **`internal/dbschema`**: idempotent `can_relay INTEGER DEFAULT 1`
migration on `observers`, plus `AssertReady` probe (server fatal-logs if
absent). Mirrored in `cmd/ingestor/db.go` `CREATE TABLE` for fresh DBs.
Annotated `PREFLIGHT: async=true` — `DEFAULT 1` is constant so SQLite
does this as a metadata-only schema rewrite.
* **`cmd/ingestor`**: `extractObserverMeta` accepts `repeat` as bool,
case-insensitive string (`on|off|true|false|yes|no`), or numeric `0|1`.
Missing field → `nil` → `COALESCE` preserves the existing column value
(back-compat with legacy observers). Plumbed through `UpsertObserverAt`
and the prepared upsert statement.
* **`cmd/server`**: `GetNonRelayObserverPubkeys` + new
`prefixMap.markNonRelay` drop matching candidates inside
`pm.resolveWithContext` at the top of the resolver, so all 4 tiers see
the pruned candidate set. `ObserverResp.CanRelay` is surfaced on
`/api/observers` and `/api/observers/{id}`. `GetNodeHealth` enriches
per-observer rows with `can_relay` so the node-detail badge renders.
Probe-and-fall-back when the `can_relay` column is absent (legacy test
fixtures).
* **`public/`**: listener vs repeater pill on observers list, observer
detail `Relay` stat card, and node-detail `Heard By` table. CSS uses
existing theme vars.
## Test
Added `TestResolveWithContext_ExcludesNonRelayObservers_Issue1290` in
`cmd/server/resolve_non_relay_1290_test.go` covering all three required
cases:
* `repeat:off` pubkey → not a candidate (assertion failed in red commit
`5f7fdb96`, passes after green `f12911dc`)
* `repeat:on` pubkey → still a candidate (regression guard)
* legacy obs (no field) → still a candidate (back-compat)
Red→green proof:
```
$ git log --oneline origin/master..HEAD
f12911dc feat(#1290): exclude listener-only observers from path-hop disambiguator
5f7fdb96 test(#1290): red — assert listener-only observers excluded from path-hop candidates
```
Full server + ingestor + dbschema + migrate test suites pass locally.
## Acceptance checklist (from #1290)
* [x] Ingestor parses `repeat` field (boolean OR string `on|off`)
* [x] Field persisted on `observers` table (new `can_relay BOOLEAN`
column, idempotent migration via `internal/dbschema`)
* [x] Server's disambiguator (`pm.resolveWithContext`) excludes
`can_relay=false` observer-nodes from path-hop candidate set
* [x] UI badge on observers list + node detail page indicating
"listener" vs "repeater"
* [x] Backward compat: legacy observers default to `can_relay=true`
* [x] Test: `repeat:off` → NOT a candidate
* [x] Test: `repeat:on` → IS a candidate
* [x] Test: legacy → IS a candidate
## Out of scope (preserved per issue)
Backfilling already-resolved paths is left as a follow-up. No
firmware/broker changes.
---------
Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local>
Co-authored-by: openclaw-bot <bot@openclaw>
This PR replaces the strict, hardcoded limits on API list endpoints
(introduced in the recent security patch) with a new
operator-configurable `listLimits` block. This change is needed as issue
1540's implementation introduced a 500max node limit on the live map or
any other function that leverages the api/nodes backend.
Previously, we attempted to bypass public caps for internal UI requests
using a heuristic based on browser headers (`Sec-Fetch-Site`). Following
review, we decided to drop that heuristic entirely to eliminate any
security-by-browser-convention surface area.
Instead, `queryLimit()` returns to its original, mathematically simple
bounds-checking shape, and the absolute maximums are now drawn from
`config.json`. This provides equal DoS protection against all callers
while allowing server operators to tune the ceilings based on the size
of their mesh (e.g. embedded devices can tighten the knobs, regional
hubs can raise them).
### Changes Made:
- **`config.go`**: Introduced a `ListLimits` config struct containing
`PacketsMax`, `NodesMax`, `AnalyticsMax`, and `ChannelMessagesMax`.
Added safe initialization to ensure default caps (10000, 2000, 200, 500
respectively) apply even if the block is omitted from the config.
- **`clamp_limit.go`**: Deleted `isInternalUIRequest` entirely and
restored `queryLimit` to its original signature (`r, def, max`).
- **`routes.go`**: Replaced all hardcoded integer ceilings on list
endpoints (`/api/packets`, `/api/nodes`, etc.) with
`s.cfg.ListLimits.*`.
- **`config.example.json`**: Added the `listLimits` block with
documentation to guide new operators.
- **`clamp_limit_test.go`**: Purged all header-heuristic testing.
### Verification:
- All 611 backend unit tests pass (`npm run test:unit`).
- Bounds-checking math continues to enforce hard DoS clipping exactly at
the operator's specified configuration limit.
---------
Co-authored-by: mc-bot <bot@openclaw.local>
Co-authored-by: openclaw-bot <bot@openclaw>
## What
Three of the four P0s from #1481's scale-test findings. Each cuts a
distinct
hot path; together they target /api/observers,
/api/analytics/neighbor-graph,
and /api/observers/{id}/analytics — the top three live offenders.
### P0-1: 5-min atomic-pointer cache for default neighbor-graph response
- Live p95 10.8s on the most-trafficked organic endpoint.
- Background recomputer (5-min cadence per operator directive) builds
the
default-filter (`minCount=5 minScore=0.1`, no region, no role)
`NeighborGraphResponse` and stores it via `atomic.Pointer`.
- `handleNeighborGraph` short-circuits on the default shape; non-default
filters take the extracted `computeNeighborGraphResponse` path
(identical
semantics to the previous inline build).
### P0-2: cache parsed `StoreObs.Timestamp` + drop RLock window
- `handleObserverAnalytics` re-parsed the RFC3339 timestamp three times
per observation, for 60k+ observations per active observer, under
`s.store.mu.RLock` — blocking writers for the full scan.
- `StoreObs.ParsedTime()` parses once via `sync.Once` (mirrors
`StoreTx.ParsedDecoded`).
- Handler snapshots the `byObserver[id]` pointer slice, releases the
RLock immediately, then iterates locally.
### P0-3: 30s cache for `/api/observers` + sargable `IN` + covering
index
- Three SQL queries on every request → ~1.7s p50 at 50-concurrent.
- Atomic-pointer 30s cache for the default (no-filter) query.
- `GetNodeLocationsByKeys` drops `LOWER(public_key) IN (...)`
(non-sargable);
callers pre-lowercase in Go and the plain `IN` matches the existing
`public_key` index.
- New ingestor migration `obs_observer_ts_idx_v1` adds composite index
`idx_observations_observer_idx_timestamp(observer_idx, timestamp)` so
`GetObserverPacketCounts` can resolve its GROUP-BY + range filter from
the index without scanning the 1.9M-row observations table.
### P0-4: deferred
`perfMiddleware`'s global mutex was claimed to serialize every API
request.
A direct test (`50 concurrent requests through the middleware, handler
sleeps 20ms each`) shows total elapsed ≈ 25ms, not 1s — the lock is held
only for the post-handler bookkeeping (a few µs). Real impact is below
measurement noise. Skipping to avoid invasive churn on PerfStats
consumers
without a demonstrable win.
## Test plan
Red → green per P0:
- `observers_cache_test.go` — handler reads `s.observersCache` before
SQL,
TTL boundary, atomic.Pointer (no mutex contention).
- `storeobs_parsedtime_test.go` — parses three timestamp shapes, caches
result, no race under concurrent readers.
- `neighbor_graph_cache_test.go` — handler serves from atomic pointer
when set, bypasses cache when `?region=` (or any non-default filter)
is passed.
Full server + ingestor suites pass: `go test -count=1 ./...`.
## Perf proof
Before/after p50/p95/p99 (50 requests × 50 concurrent) against prod
(before)
and staging once CI deploys (after) will be posted as a PR comment per
the
operator's "no merge without proof of improvement" gate.
Closes#1481
## TDD exemption — P0-1 and P0-2 (net-new surfaces, AGENTS.md)
Per CoreScope `AGENTS.md` § "Exemptions": **net-new code surfaces with
no
prior tests to break** may land tests in the same PR without a strict
test-first → impl commit split.
- **P0-1 (neighbor-graph atomic-pointer cache)** — `neighborGraphCache`,
`recomputeNeighborGraphCache`, `loadNeighborGraphCacheBytes`,
`startNeighborGraphRecomputer` and the default-shape short-circuit in
`handleNeighborGraph` were brand-new code with no pre-existing
assertions covering them. There was no green test to first turn red.
- **P0-2 (cached `StoreObs.Timestamp` + RLock window drop)** —
`StoreObs.ParsedTime()` and the snapshot+release pattern in
`handleObserverAnalytics` were new surfaces; the prior code did the
parse inline per call with no behavioural test to break.
P0-3 was authored properly red-then-green (commit `6e63ec6a` red, then
`83ae129b` green) and does NOT use this exemption.
## Default-filter detection vs frontend reality (#1483 follow-up)
The Neighbor Graph analytics tab in `public/analytics.js` fetches
`/analytics/neighbor-graph?min_count=1&min_score=0` because the
client-side sliders need the full edge set to filter from. That shape
did NOT match the `(5, 0.1)` cached default, so the UI tab still paid
the cold compute cost despite #1481 P0-1.
The #1483 follow-up commit caches BOTH shapes in the same recomputer
pass:
- `(minCount=5, minScore=0.1, no region, no role)` — `live.js`
affinity-scoring consumer.
- `(minCount=1, minScore=0, no region, no role)` — analytics tab.
Both are served from `atomic.Pointer` with an `X-Cache-Age-Seconds`
header. The per-shape cost in the background goroutine is roughly
linear in edge count; total recompute time stays well under the
5-minute cadence on prod-scale graphs.
---------
Co-authored-by: openclaw-bot <bot@openclaw.dev>
Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
## Summary
Issue #1478 — surface observers whose envelope timestamps are being
clamped because they're emitting zone-less local-time strings (UTC-N
observers showed up perpetually as "Stale" before #1466, and per-packet
rxTime is still clamped to ingest time for them, muddying
propagation-delay analytics).
Now the UI tells operators which observers are misconfigured + how to
fix it.
## What changed
### Ingestor (cmd/ingestor)
- New `observers_clock_naive_v1` migration adds three columns to
`observers`:
- `clock_skew_seconds INTEGER` (signed: negative = behind UTC, positive
= ahead)
- `clock_skew_count_24h INTEGER` (rolling 24h event count)
- `clock_last_naive_at TEXT` (RFC3339 timestamp of last clamp)
- `resolveRxTime` now returns `(rxTime, naiveSkewSec)`. The
packet-handler call site invokes `store.RecordNaiveSkew(observerID,
deltaSec)` whenever a naive envelope is clamped (the existing >15 min
naive-tolerance path). The counter resets to 1 if no event in the prior
24h, else increments. Single INSERT-or-UPDATE round trip per clamp.
### Server (cmd/server)
- `Observer` struct + `GetObservers` / `GetObserverByID` extended to
scan the three new columns.
- `ObserverResp` gains four JSON fields exposed by `/api/observers` and
`/api/observers/{id}`:
- `clock_naive` (bool, derived from `clock_last_naive_at` being within
24h)
- `clock_skew_seconds`, `clock_skew_count_24h`, `clock_last_naive_at`
- Decay is **read-side**: a stale event yields `clock_naive=false` with
zero counts. No background sweep, no writes from the read-only server,
no race with the ingestor.
### Frontend (public)
- `window.ObserversNaiveChip.render(o)` — total render helper, returns
⚠️ chip HTML when `o.clock_naive===true`, `""` otherwise. Used inline in
the observers-list `name` cell and in the row-detail slide-over. Tooltip
explains magnitude + direction + count + fix.
- `window.ObserverDetailNaiveBanner.render(obs)` — yellow alert banner
at the top of the observer-detail page with the skew magnitude,
last-event timestamp, and the actionable fix ("Set host clock to UTC, OR
emit Z-suffixed/offset-aware timestamps from the observer script").
## TDD trail
- `5ddd5b42` red: backend `cmd/server/observer_naive_clock_1478_test.go`
(3 tests asserting JSON fields + 24h decay) + frontend
`test-observer-naive-clock-1478.js` (8 jsdom-style tests asserting
helpers exist and render correctly). Both failed on master with
field-missing / export-missing assertions.
- `4ecc79c8` green backend: schema + Observer / GetObservers /
ObserverResp / handler decay.
- `2137ab81` green frontend: chip + banner helpers and call sites.
## Tests
- `cd cmd/server && go test ./...` → all green (full suite, 46s)
- `cd cmd/ingestor && go test ./...` → all green (full suite, 98s)
- `node test-observer-naive-clock-1478.js` → 8/8 pass
- `node test-frontend-helpers.js` → unchanged from master (pre-existing
failures only)
## Acceptance (issue #1478)
- ✅ Observer running with `python datetime.now().isoformat()` (naive,
off by N hours) → `clock_naive=true` after the next clamp → UI shows ⚠️
chip + banner.
- ✅ Observer with `datetime.now(timezone.utc).isoformat()` (Z-suffixed)
→ never clamped → never flagged.
- ✅ Observer that fixed its clock → `clock_naive` returns to `false` 24h
after the last clamp event (read-side decay).
Closes#1478.
---------
Co-authored-by: openclaw <bot@openclaw.local>
## Summary
Follows the reconciliation recommendation in #916 — extracts only the
NET-NEW persistence layer from that PR (which is now superseded by #1002
for the overlay UI) into a focused 6-file change against current master.
**What this adds:**
- `multibyte_sup_v1` migration: `multibyte_sup INTEGER NOT NULL DEFAULT
0` + `multibyte_evidence TEXT` on `nodes`/`inactive_nodes` so capability
survives restart
- `hasMultibyteSupCols` schema detection gates the persist/load paths
- `loadMultibyteCapFromDB()`: pre-populates `mbCapSnapshot`/`mbCapIndex`
at startup — cold starts serve last-known capability without waiting for
the first ~15s analytics cycle
- `maybePersistMultibyteCapability()` + `persistMultibyteCapability()`:
after each analytics cycle; TryLock-gated (concurrent cycles coalesce);
skips `sup==0` entries (data-destruction guard)
- `GetMultibyteCapFor(pk)`: O(1) map lookup; both `handleNodes` and
node-detail call sites updated from the O(N)-alloc
`GetMultiByteCapMap()`
**What this explicitly does NOT change:**
- API field names (`multi_byte_status`, `multi_byte_evidence`,
`multi_byte_max_hash_size`)
- `EnrichNodeWithMultiByte` — unchanged
- `GetMultiByteCapMap` — still present for any external callers
- `public/map.js`, `public/live.css`, `Dockerfile`, `docs/` — zero
frontend churn
## Test plan
- [x] `TestMultibyteCapPersistRoundTrip` — confirmed values survive
persist → fresh-store load
- [x] `TestMultibyteCapPersistSkipsUnknown` — data-destruction guard:
`sup==0` entry does not overwrite DB-confirmed value
- [x] `TestMultibyteCapMaybePersistCoalesces` — TryLock coalesces 10
concurrent callers without deadlock
- [x] `TestMultibyteCapGetMultibyteCapForO1` — O(1) index returns
correct entry / false for unknown pubkey
- [x] `TestMultibyteCapLoadFromDB` — only `sup>0` rows loaded; `sup==0`
row excluded
- [x] `TestSchemaMultibyteSupColumns` — migration adds columns to both
tables; idempotent on second `OpenStore`
- [x] All existing `TestMultiByteCapability_*` tests pass unchanged
- [x] Full ingestor test suite: `ok` in 27s
- [x] `go build ./cmd/server/ && go build ./cmd/ingestor/` clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: openclaw-bot <bot@openclaw>
## Summary
Fixes#1345 — the packets page shows "no recent activity" while MQTT
ingest is healthy because the default `/api/packets` query was `ORDER BY
first_seen DESC`, and PR #1233 redefined `first_seen` as the observer's
radio receive time (rxTime). When an observer buffers offline and
uploads hours later, its packets land with hours-old `first_seen`
values; older-ingested packets with fresher rxTime then crowd the top of
the list and the visually freshest activity disappears.
## Fix
Switch the default ordering to `t.id DESC` (ingest order) on
`/api/packets` and the closely-related endpoints. `id` is monotonic with
ingest time and immune to buffered uploads.
Endpoints changed (all use the same fix for the same reason):
| Path | Function | File |
|------|----------|------|
| `GET /api/packets` (default) | `DB.QueryPackets`, `Store.QueryPackets`
| `cmd/server/db.go`, `cmd/server/store.go` |
| `GET /api/packets?nodes=…` | `DB.QueryMultiNodePackets`,
`Store.QueryMultiNodePackets` | same |
| Node detail "recent transmissions" |
`DB.GetRecentTransmissionsForNode` | `cmd/server/db.go` |
## `since=` semantic — preserved
`since=` still filters by `first_seen` (RFC3339 path uses the
observations.timestamp subquery), i.e. "packets the network received
since X." Buffered uploads of older packets are still excluded from a
`since=15m` view even if they were ingested in the last 15 minutes. Only
the **display order** changes; filtering by receive time is unchanged.
## Audit — NOT changed
- `Store.QueryGroupedPackets` already sorts by `LatestSeen` (max
observation timestamp), which is correct for the grouped view and immune
to the buffered-upload regression.
- `GetChannelMessages` and channel `sample_json` subqueries keep
`first_seen DESC` — channel message chronology is meaningful for message
UX; if buffered uploads become a problem here too it's a separate UX
call (out of scope for #1345).
- `s.packets` insertion ordering (Load + ingest) — untouched. The fix
sorts at query time so we don't perturb `oldestLoaded` invariants.
## Tests — TDD red → green
- Red: `508f4371` adds `cmd/server/packets_order_test.go` with two cases
— order assertion (failed on master with `[fresh, buffered]`) and
since-filter semantic (RFC3339 path uses observation timestamps).
- Green: `0fd685e7` switches the SQL + in-memory ordering. Tests pass;
full `cmd/server` suite green locally (44s).
## Out of scope
- Re-thinking #1233's first_seen semantics
- Adding a UI sort toggle (issue's option 2)
- Channel-message page ordering
## Preflight
Clean (`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh
origin/master`).
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
Red commit: 702d82eb5e (CI: see Actions
tab for fix/issue-1366)
## What
Channel view emits the max observation timestamp (`tx.LatestSeen`)
instead of the analyzer's first-observation time (`tx.FirstSeen`) as the
rendered `timestamp` field. A new `first_seen` field is exposed
alongside for debug surfaces. `sender_timestamp` continues to be
returned in the JSON response but is intentionally NOT used as the
rendered time (client clocks are unreliable).
## Root cause
Two parallel call sites both emitted the wrong field:
- `cmd/server/store.go` — `GetChannelMessages` (~line 4807): set
`entry.Data["timestamp"] = strOrNil(tx.FirstSeen)` for every new dedup
entry. `tx.FirstSeen` is the analyzer's first-ever observation time of a
`transmissions.hash` row; for heartbeat-style packets (e.g. `BlorkoBot
🤖` posting the same status line periodically), the hash is stable, so
FirstSeen stays pinned at the very first observation while the message
keeps retransmitting hours later. Operator sees "old" message timestamps
for live messages.
- `cmd/server/db.go` — `GetChannelMessages` (~line 1757): same problem
against the SQLite-backed query path. Used `nullStr(fs)` (where `fs` is
`t.first_seen`) for the `timestamp` field.
### Repro from staging
Same packet, same hash `aba4f0493249de57`, sender `BlorkoBot 🤖`:
- `/api/channels/%23test/messages` → `timestamp: "2026-05-25T15:53:20Z"`
(FirstSeen, 7h+ in the past)
- `/api/packets?hash=aba4f0493249de57` → `first_seen:
"2026-05-25T22:53:19Z"` (latest obs), `observation_count: 84`
The packets view used max-obs correctly; the channels view did not. 7h
gap matches operator screenshot.
## TDD red → green
Red: `cmd/server/channels_message_order_1366_test.go` — three tests:
- `TestChannelMessages_TimestampUsesLatestSeen`: seeds a CHAN tx with
observations 7h apart, asserts returned `timestamp` ≈ latest observation
epoch (±1s). Fails under FirstSeen with Δ=−25200s.
- `TestChannelMessages_TimestampNotSenderTimestamp`: seeds a CHAN tx
whose decoded `sender_timestamp` is year-2000 (bad RTC). Asserts the
rendered `timestamp` parses to current year — guards against the
tempting "just use sender_timestamp" alt-fix that would let bad client
clocks corrupt the view.
- `TestChannelMessages_TimestampIsUTCZ`: asserts the emitted string is
unambiguously UTC (suffix `Z` or `+00:00`) so browsers don't apply a
local-zone shift.
Green commit changes:
- `store.go`: emit `tx.LatestSeen` (with FirstSeen fallback if no obs);
add `first_seen` field.
- `db.go`: join `o.timestamp` per-observation, track max epoch per tx,
emit RFC3339 UTC at the end; add `first_seen` field.
`sender_timestamp` remains in the response — unchanged shape, frontend
never read it for the rendered time (verified: only `msg.timestamp` is
consumed in `public/channels.js:1902`).
## Manual verification (post-merge)
1. Deploy to staging.
2. Curl `/api/channels/%23test/messages?limit=5` and
`/api/packets?hash=<recent>`. The channel `timestamp` field MUST equal
the packets `first_seen` (max obs) for the same hash, NOT lag it.
3. Send a fresh GRP_TXT via a MeshCore client into a watched channel.
Within 15s, refresh the Channels view at `/channels`. The new message
MUST render at the bottom with the correct (current) time.
## Why not `sender_timestamp`?
It's a per-client field, decoded from the payload. Many MeshCore
firmware builds run without RTC/NTP/GPS and report bogus values.
Trusting it for display would propagate bad client clocks into the
analyzer UI — the analyzer is the source of truth for UTC, not the
client.
Fixes#1366
---------
Co-authored-by: CoreScope Bot <bot@corescope>
Co-authored-by: bot <bot@kpa-clawbot.dev>
Co-authored-by: openclaw-bot <bot@openclaw.local>
## Summary
- Adds configurable GPS polygon areas to `config.json`; nodes are
attributed to an area if their last-known position falls inside the
polygon
- New `Area: …` dropdown filter (matching the existing region filter
style) appears on all analytics, nodes, packets, map, and live screens
when areas are configured
- Backend resolves area membership with a 30s TTL cache; area filter
bypasses the 500-node cap on `/api/bulk-health` so all area nodes are
always returned
- Includes a polygon builder tool (`/area-map.html`) for drawing and
exporting area boundaries
## Changes
**Backend**
- `AreaEntry` type + `Areas` config field
- `GetNodePubkeysInArea` DB query + `resolveAreaNodes` (30s TTL,
`areaNodeMu` RWMutex)
- `PacketQuery.Area` + `filterPackets` polygon check
- `?area=` param propagated through all analytics, topology,
clock-health, and bulk-health routes
- `/api/config/areas` endpoint
**Frontend**
- `area-filter.js`: single-select dropdown, persists to localStorage,
cleans up stale keys on load
- Wired into analytics, nodes, packets, channels, map, and live pages
- Live map clears node markers on area change
**Docs & tools**
- `docs/user-guide/area-filter.md` — configuration and usage guide
- `docs/api-spec.md` — updated with new endpoint and `?area=` param
table
- `tools/area-map.html` — polygon builder for defining area boundaries
- Demo areas added to `config.example.json`
## Test plan
- [x] No areas configured → filter dropdown does not appear on any page
- [x] Areas configured → dropdown appears, "All" selected by default
- [x] Selecting an area filters nodes/packets/topology/map correctly
- [x] Selecting "All" restores unfiltered view
- [x] Selection persists across page reloads (localStorage)
- [x] Stale localStorage key (area removed from config) is cleared on
load
- [x] `/api/bulk-health?area=X` returns all nodes in area (no 500-node
cap)
- [x] `/api/config/areas` returns correct list
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
@
## What this PR does
Implements region-scoped transport-route packet tracking with two
sub-features:
### Feature 1 — Scope statistics (`scope_name`)
- At ingest, transport-route packets (route_type 0/3) with Code1 !=
`0000` are HMAC-matched against configured `hashRegions` keys (mirroring
the `hashChannels` pattern). Matched region name (or `""` for unknown)
stored in new `transmissions.scope_name` column via migration
`scope_name_v1`.
- New `GET /api/scope-stats?window=` endpoint (1h/24h/7d, 30s
server-side TTL) returning transport totals, scoped/unscoped counts,
per-region breakdown, and time-series.
- New **Scopes** tab in Analytics with summary cards, per-region table,
and two-line SVG chart. Auto-refreshes every 60s.
### Feature 2 — Node default scope (`default_scope`)
- Per-node `default_scope` column on `nodes`/`inactive_nodes` (migration
`nodes_default_scope_v1`) tracks the most recently matched region for
each node, derived from transport-scoped ADVERT packets.
- `GET /api/nodes` response includes `default_scope` field when column
is present.
- Node detail panel displays the default scope badge.
- Async startup backfill (`BackfillDefaultScopeAsync`) populates the
column for nodes with pre-existing ADVERT data.
### Config
Add `hashRegions` to `config.json` (see `config.example.json`). One
entry per region name (with or without leading `#`).
@
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
