Three follow-up improvements to the View Path map, all requested
together:
- A compact color/symbol legend (route color, approx marker, first-
heard ring) replaces the old prose paragraph explaining the same
things -- shorter intro text, faster to scan.
- A "show only farthest-traveled route" checkbox appears whenever a
packet was heard by more than one station, to declutter busy
20+-branch packets. Only added/removed as a Leaflet layer toggle;
the primary branch's own markers/polyline are never touched.
- Every touched area is now shaded directly on the map (a real
polygon when the area was drawn as one, a rectangle fallback for
bbox-only areas) instead of only listed as text -- ties the
"touched: X, Y" footer line to actual geography. Non-interactive so
shapes never steal a click meant for a branch marker.
The shading needed PacketPathResponse.TouchedAreas to carry geometry,
not just labels -- changed from []string to []TouchedAreaShape
(label + polygon or bbox). annotatePacketPathTouchedAreas now looks
up the matched area's full config entry via AreaKeyForPoint instead
of just its label via AreaForPoint.
The ping-bot reply's "touched" list caps at 3 to keep a chat bubble
readable, but View Path's map has room to show the full set: every
configured area any point or observer on the path resolved to,
deduped and alphabetized.
No extra DB round-trip needed here (unlike the pong reply's version)
-- GetPacketPath already resolved every position, including the
neighbor-centroid approximation fallback, so annotatePacketPathTouchedAreas
just reads the lat/lon already on the response.
The "relays for 2+ distinct region scopes" definition doesn't hold up
in practice: most nodes end up carrying both a broad national/regional
scope (e.g. #dk, #eu) and several local ones, so the vast majority of
repeaters would eventually qualify as "bridge" -- the flag stops
meaning anything useful.
Removes IsBridge from PacketPathPoint/PacketPathObserver, the
markBridgeRepeaters handler step, the purple-outline styling, and the
dedicated test file -- keeps PublicKey on PacketPathObserver, since
the click-to-node-detail feature depends on it independently.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DistanceFromFirstKm to PacketPathBranch: the great-circle
distance (haversine) between a branch's own Observer and First's
Observer. Zero for First itself. Deliberately omitted when either
side is positioned via Approx (a neighbor-centroid estimate) -- a
distance computed against a guess isn't a real measurement worth
surfacing.
Rendered in the observer tooltip as "42.3 km away" alongside the
elapsed-time label, giving a concrete sense of how far the flood
physically reached, not just how many hops or how long it took.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds IsBridge to PacketPathPoint/PacketPathObserver and PublicKey to
PacketPathObserver (needed to look bridges up, and useful on its own
for future features). Set by a new markBridgeRepeaters handler-level
step in routes.go, not GetPacketPath itself, since the underlying data
(TransportedScopes per repeater) lives in the in-memory store, not
SQL -- same "relays for 2+ distinct regions" definition and data
source as the Foreign Traffic tab's existing Bridge badge
(ScopeStatsResponse.bridgeRepeaters), just applied to whichever
handful of pubkeys one packet's path touches.
The map gives a bridge repeater a bold purple outline (on top of
whatever primary/approx styling already applies) plus a tooltip note,
so the mesh's backbone nodes stand out regardless of which branch
they're in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PacketPathPoint already carried Role; PacketPathObserver now does too
(from the observer's own nodes row, when it's known as a mesh node
itself and not just an MQTT/API listener). Rendered as a small icon
prefix in the tooltip (📡 repeater, 🏠 room, 📱 client, 🌡️ sensor) --
markers stay plain circleMarker dots throughout, since a role-specific
shape would clash with the color/dash coding already carrying primary,
approx, and observer meaning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nearestPositionedNeighbor now also returns how many positioned
neighbors fed the weighted centroid and the widest distance between
any two of them (0 with a single contributor). Exposed via new
ApproxNeighborCount/ApproxSpreadKm fields on PacketPathPoint and
PacketPathObserver.
The map scales the approximate marker's size/opacity accordingly: one
neighbor (or several that disagree widely) renders as a bigger,
fainter ring; several agreeing neighbors render tighter and more
solid. The tooltip also states the contributor count.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds SecondsAfterFirst to PacketPathBranch: the gap between the
earliest-arriving observation (First) and this branch's own deepest
observation. Zero for First itself. Rendered in the observer's tooltip
as "+4.7s" (or "Nm Ss" for longer gaps, "first to arrive" for zero) --
gives a sense of the propagation order across the flood, not just
depth/breadth.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nearestPositionedNeighbor picked exactly one neighbor (the strongest by
edge count) and used its exact coordinates -- so the approximate marker
always landed precisely on top of that neighbor's own dot. Since each
neighbor's own position is a real, precise fix, and only the
unpositioned node's position relative to them is unknown, more
positioned neighbors should narrow the estimate rather than being
ignored in favor of just the top one.
Now sums every positioned neighbor's coordinates weighted by
neighbor_edges.count and returns the centroid. With exactly one
positioned neighbor this is unchanged (reduces to that neighbor's exact
position); with several, the marker settles somewhere among them
instead of overlapping a single specific node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A hop or observer with no position of its own (no self-advertised GPS,
no name match, no IATA) previously stayed unplotted entirely. Adds a
last-resort fallback: look up its strongest neighbor_edges neighbor
(ranked by observation count) and, if that neighbor has a real
position, borrow it as an approximate stand-in -- flagged via a new
`approx` field so callers never mistake it for a real fix.
The View Path map renders approximate markers hollow and dashed
instead of solid, with a status-line callout ("N approximate (via
nearest neighbor)"), and the "N hops without a known position" count
drops accordingly for whichever ones now have a stand-in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetPacketPath's branches are sorted deepest-first, so the map had no
natural "where did this start" anchor. Adds a `first` field: the
single earliest-arriving observation across every station regardless
of observer or hop depth (the same "first observation wins" pick
already used for a ping message's own meta line), positioned the same
way as any other observer (own GPS, then name match, then IATA).
The map draws it as a green ring on top of everything else, since it
usually coincides with one of the already-plotted branch dots, plus a
status-line callout -- an approximate landmark for where the message
entered the visible mesh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetPacketPath only ever tried the hardcoded iataCoords table to place
an observer. Any station whose configured IATA code isn't a real
airport (a custom/regional code, or a typo) fell out of the map
entirely -- even when the station is itself a mesh node that has
self-advertised a real GPS position, the same source /api/observers
and the Wardriving tab already treat as authoritative.
Now checks the observer's own node-table position first (folded into
the existing bulk pubkey lookup, no extra query) and only falls back
to the IATA table when it has none.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetPacketPath used to resolve only the single farthest-traveled
observation and discard the rest. It now returns one branch per
distinct station (kept at that station's own deepest observation),
each with its own hop count and, where resolvable, relay chain --
so the map visualizes the packet's full flood spread instead of one
route. Stations whose path never resolved, or who heard the packet
directly (0 hops), still contribute a branch via their own position,
rather than being silently dropped.
The "View path" link in the ping-bot reply no longer requires hops > 0
to show, since even an all-direct ping still has observer spread worth
plotting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/packets/{hash}/path resolves a packet's DEEPEST observation
(same "farthest leg is more informative" reasoning as the ping-bot reply
itself) to a geographic point sequence: each relay's name/role/lat/lon in
path order, plus the hearing observer's position (from its configured
IATA code, like the Wardriving tab). Hops that have never advertised a
GPS position come back with null lat/lon rather than being dropped, so
the frontend can draw a gap instead of guessing.
Frontend: public/packet-path-map.js is a small on-demand Leaflet modal
(reuses node-reach-map.js's tile/marker conventions, but draws an ordered
chain instead of a star) opened via a new "View path" link on the
ping-bot reply -- shown only when there's an actual multi-hop route and
packet hash to look up. Kept general (keyed by packet hash, not
ping-specific) since any packet with a resolved path could use it later.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MeshCore firmware sends two capability bytes on ADVERT packets (wire bits
per AdvertDataHelpers.h) whenever the HasFeat1/HasFeat2 flags are set.
CoreScope already decoded them per-packet but discarded them rather than
storing a per-node value.
- New feat1/feat2 columns on nodes/inactive_nodes (internal/dbschema,
idempotent ALTER like the existing multibyte_sup/multibyte_evidence
columns).
- Ingestor's UpdateNodeTelemetry now writes feat1/feat2 alongside
battery_mv/temperature_c in the same COALESCE-based UPDATE, from the
same ADVERT payload.
- Server exposes them on /api/nodes and /api/nodes/{pubkey} (nullable,
same shape as battery_mv/temperature_c).
- Node detail page shows them as raw hex (0x....) in the Overview panel
when present -- undecoded, since CoreScope doesn't know the individual
bit meanings, but no longer silently discarded.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GetHopDepthAnalytics now buckets its scoped/unscoped hop-index tallies by
time (same 5min/1h/6h bucketing as GetScopeStats' TimeSeries) and computes
a per-bucket median for each series, exposed as a new TimeSeries field.
Median is a *int, nil when a bucket had no traffic of that kind -- 0 is a
valid median hop, so absence has to stay distinguishable from zero.
Frontend renders it as a two-line SVG trend chart on the Scopes > Hop
Depth sub-tab (same visual language as Overview's scoped/unscoped
time-series), answering "is containment getting better or worse over the
window" rather than just a single window-wide snapshot. Each series'
polyline is built from contiguous non-null segments so a gap renders as
a visible break instead of silently interpolating through missing data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the #1812 per-node relay hop-count work with a network-wide view:
GET /api/analytics/hop-depth answers (1) whether scoped traffic actually
travels fewer hops than unscoped before hitting a repeater's flood.max cap
(Scopes tab Overview: new "Flood Containment" comparison), and (2) which
repeaters relay unscoped traffic that already traveled far vs merely
locally (Foreign Traffic tab: min/median/max hop columns joined onto the
existing unscoped-relay table by public key).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/nodes/{pubkey}/hop_analytics?days= returns, for each recent
transmission that passed through this node as a relay, the node's own
0-based index within the packet's resolved path -- the value MeshCore
firmware compares against flood_max/flood_max_advert/flood_max_unscoped
in allowPacketForward (Packet.cpp path_len = header & 63; MyMesh.cpp
getPathHashCount() >= flood_max). Deliberately distinct from the
existing hopDistribution field on /analytics, which measures path
length to whichever observer reported the packet -- a different,
unrelated number, confirmed via the upstream issue's multi-round
firmware-source discussion.
Candidate-finding reuses the same byPathHop-index + resolved_path
confirmation approach as handleNodePaths (routes.go), written fresh
rather than refactored out of that function to avoid any regression
risk in its older, correctness-critical prefix-collision handling
(#929, #1197, #1278, #1352). Only transmissions with a canonical
resolved_path contribute a data point -- no resolved_path means no
reliable hop index, so it's skipped rather than guessed.
Frontend: new "Relay Hop-Count" card on the node analytics page --
histogram (Chart.js) + a hand-drawn boxplot canvas above it sharing the
same x-axis (no Chart.js boxplot plugin is loaded), filterable by chip
matching the firmware knob names (flood default, flood_advert,
flood_unscoped, direct).
?hasScope=true|false keeps only nodes that have/haven't ever
transported a region-scoped packet; ?hashRegion=eu,be (comma-separated,
leading # optional) keeps only nodes that have transported at least
one of the given regions. Both combine with each other (AND) and with
the existing ?role= filter, matching the suggested API shape from
https://github.com/Kpa-clawbot/CoreScope/issues/1862.
Implemented as two new predicates in the existing nodeListPostFilters
(same post-SQL-LIMIT compensation-loop machinery geo_filter/blacklist/
area already use), backed by the same bulk relay-info map
(GetRepeaterRelayInfoMap) the Scopes tab's "Repeaters Never Relaying
Any Scope" section reads -- fetched once per request, not per DB page.
Adds AirtimeMs to WardrivingSession: total network airtime consumed
relaying a session's messages, using the exact same formula as the
Overview tab's "Relay Airtime Share" (issue #1768) — ToA(payload_bytes)
x COUNT(DISTINCT resolved repeater in path), summed per message.
That formula needs the in-memory store's resolved-path index
(resolvedPubkeyReverse), which db.go's SQL-only wardriving code
doesn't have — so buildWardrivingSessions now also tracks each
session's transmission IDs (unexported, json:"-"), and the route
handler calls the new PacketStore.AirtimeForTransmissions(txIDs) to
fill in AirtimeMs after the fact. Omitted entirely (nil) in DB-only
mode rather than shown as a misleading zero.
Frontend adds an Airtime column to the Sessions table (ms/s
formatting, dash when unavailable).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found in production: at least one wardriving client appends plaintext
"<lat>,<lon>" after the standard token (e.g.
"MM:c3e_zJ1rUA:55.59743,13.00128"), confirmed against live traffic.
The generic anomaly framing (flagging non-standard payload length,
showing a raw hex dump) was based on the wrong assumption that this
was an undocumented binary format — it's actually deliberate,
human-readable coordinate sharing by that sender's client.
Replaces WardrivingAnomaly/detectWardrivingAnomalies/
StandardPayloadCount with WardrivingGPSShare/detectWardrivingGPSShares:
detects the specific "<token>:<lat>,<lon>" suffix (with range
validation) and reports the most recent position per sender. The
per-message drill-down gets Lat/Lon instead of a standard/anomaly
flag. Frontend replaces the "Payload Anomalies" section with "GPS
Sharing", linking each position to the live map
(#/map?lat=..&lon=..&zoom=15, an existing deep-link format).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/analytics/wardriving/sender-messages returns one sender's
individual messages (most-recent-first, capped at 200): resolved
entry-point path (path[0] first), per-observer SNR/RSSI, and payload
standard/anomaly classification. Pass since+until to scope to one
session's exact range; otherwise it covers the sender's whole window.
Frontend makes sender names in Top Senders and Sessions clickable —
clicking toggles an inline expansion row with that sender's messages,
resolving every distinct path prefix in one /resolve-hops call (same
unique_prefix-only discipline as the aggregate Entry Points table).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with a payload-anomaly detector:
every "MM:<base64>" message is checked against the standard 7-byte
anonymous session token (confirmed empirically against live traffic).
Non-standard lengths or undecodable payloads are grouped per sender
as a candidate signal that MeshMapper's optional "Broadcast My
Coordinates" mode is active — surfaced as a raw hex dump, never
interpreted as lat/lon, since that mode's byte format is undocumented.
Messages without the "MM:" prefix (plain channel chat) are ignored
entirely rather than polluting either bucket.
Frontend adds a Payload Anomalies table plus a stat card, completing
all 6 originally-proposed wardriving analytics ideas.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with each sender's messages
grouped into distinct sessions/runs — a gap over 15 minutes starts a
new one. Each session reports duration, message count, and how many
distinct entry-point repeaters/observers it touched. Frontend adds a
Sessions table (most-recent-first) and a session-count stat card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends GET /api/analytics/wardriving with avgSnr/avgRssi over the
same time buckets as the activity series, plus overall averages.
Frontend adds two min/max-scaled line charts (SNR has no natural
zero floor, RSSI is negative dBm, so these can't reuse the 0-baseline
message-volume chart) and two stat cards.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/analytics/wardriving endpoint plus an Analytics tab
covering the three requested angles: activity over time + top
senders, entry-point repeaters (path[0] tally, unique_prefix-only
name resolution), and per-observer coverage using observers' known
IATA coordinates. MeshMapper's on-air ping is an anonymous session
token by default, not the sender's live GPS, so sender position
itself isn't tracked — documented in the tab and OpenAPI description.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- MAJOR: config.example.json was missing geoFilterExemptNodeList
entirely — every other Config field in this repo gets a sibling
_comment_<field> entry (established, actively-used convention
throughout the file, ~20 existing examples), documented here to
match. Also corrected the adjacent geo_filter _comment, which
predates this PR and claimed it "restricts ingestion" — it doesn't;
that's foreignAdverts.mode=drop's job, unrelated to this flag.
- MINOR: ?geoFilter=<v> only recognized the literal string "1" to
enable and treated anything else (including "true") as disabling —
a real footgun since "true" reads as an obviously valid boolean
value. Now accepts 1/true and 0/false as explicit overrides; any
other value (including absent) falls through to the deployment
default instead of silently flipping it off.
- Tightened the Foreign Traffic tab's stop-hook test: it previously
only asserted stop() doesn't throw, which a no-op stub would also
pass. Sandbox's setInterval/clearInterval are now real spies
tracking live/cleared ids, so the test verifies render() registers
exactly one interval and stop() actually clears that exact id
(idempotently on a second call).
- NIT: relays.sort now wraps unscoped_relay_count_24h in Number(...)
defensively (AGENTS.md § Type Safety — cast at the boundary).
- NIT: foreignNodes.sort now parses last_seen once per node
(decorate-sort-undecorate) instead of twice per comparator call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
7cc5aae's GeoFilterAppliesToNodeList had the polarity backwards: it
defaulted to false, meaning geo_filter would NOT apply to the node
list unless a deployment explicitly opted in — but every existing
deployment that already had geo_filter configured (predating this
field entirely) would decode that field as false too, since it's
simply absent from their config.json. They'd silently get the new
"show everything" behavior on upgrade with no way back short of
reading release notes and adding a new config key — exactly the
breaking change the opt-in gate was meant to prevent, just one layer
deeper.
Renamed to GeoFilterExemptNodeList (still defaults false): geo_filter
now applies to the node list by DEFAULT when configured, matching the
long-standing #730 behavior exactly, for every deployment predating
this field. Only a deployment that explicitly sets
geoFilterExemptNodeList=true (i.e. one adopting geo_filter fresh,
purely for foreign_advert classification/analytics) gets the
non-filtering default. ?geoFilter=0/1 still overrides either default
for a single request.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per bot review on PR #1852 (comment 5012514227): the ?geoFilter=1
opt-in change (ffb2c84) flips a public API's default behavior with no
migration path. On THIS deployment geo_filter was never configured
before this session, so nothing was ever relying on the old filtered
behavior here — but the PR targets the shared upstream codebase, and
any other deployment that already had geo_filter configured (the
pre-existing #730 declutter feature) would silently start getting an
unfiltered node list after upgrading to this code, with no way back.
Adds Config.GeoFilterAppliesToNodeList (default false, matching the
new non-surprising behavior for anyone configuring geo_filter fresh).
A deployment that intentionally relies on the old always-on filtering
sets it to true in config.json to keep exactly the behavior it had
before this PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Configuring geo_filter alone was silently changing what GET /api/nodes
returned — every node outside the polygon and not yet foreign_advert
-tagged (which only happens on that node's next ADVERT after
geo_filter is set) vanished from every view built on this endpoint,
including the live map. There was no way to see them again short of
waiting for each one to re-advertise.
geo_filter's own purposes — the ingestor tagging foreign adverts, and
the explicit prune-geo-filter admin flow — are untouched; this only
gates the passive node-list declutter view behind an explicit query
param, so turning geo_filter on to build analytics (e.g. the Foreign
Traffic tab) doesn't have the side effect of hiding nodes from the map.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds, per node, how many distinct FLOOD adverts it originated in the
last 7 days. Zero-hop adverts (route_type DIRECT) are excluded, so a
nearby observer hearing a node's cheap local adverts does not inflate
the number - the existing advert_count mixes both kinds and cannot tell
a chatty flooder (mesh-wide airtime) from
the recommended 240-minute zero-hop cadence (local only).
Consumers (the ArcScope repeater advisor) rate advert hygiene against
the community practice of one flood advert every ~49h; with the mixed
total, a correctly configured repeater looked chatty whenever an
observer sat within zero-hop range.
Implemented like the relay-liveness fields: a pure, unit-tested counter
over (first_seen, route_type, hash) entries with the same timestamp
parsing and hash dedup, fed by a from_pubkey-indexed query capped at the
2000 most recent advert rows. The flood route-type constant is named
advertRouteTypeFlood so this merges independently
of the open unscoped-relay PR (#1823).
---------
Co-authored-by: Waydroid Builder <build@waydroid.local>
## What
Adds a per-node API field `unscoped_relay_count_24h` on repeater/room
nodes: the
number of the node's 24h relay-hops that were unscoped floods
(route_type == ROUTE_TYPE_FLOOD). A strict subset of relay_count_24h.
## Why
A well-configured repeater runs `flood.max.unscoped 0` and should not
rebroadcast
unscoped floods — each one is re-sent by every repeater that hears it,
so one
packet turns into mesh-wide traffic. Exposing this lets clients (the
ArcScope
repeater advisor) detect and flag that base-config problem from observed
packets.
## How
Computed like relay_count_24h in both paths (bulk /api/nodes + per-node
detail)
with a route_type==FLOOD filter; reuses the byPathHop index, no
migration. Wired
into both handlers + OpenAPI schema + unit tests (per-node and bulk).
Co-authored-by: Waydroid Builder <build@waydroid.local>
Documented Node schema (the four #672 usefulness axes + composite + A-F
grade + relay fields) and response schemas on the node endpoints.
Documentation-only; no behaviour change. Pairs with #1762 (documents the
metrics it adds).
Co-authored-by: Waydroid Builder <build@waydroid.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds MQTT source status visibility per #1043 acceptance criteria:
- **Ingestor:** per-source counter registry
(`cmd/ingestor/source_status.go`) tracking `connected`,
`lastConnectUnix`, `lastDisconnectUnix`, `lastPacketUnix`,
`connectCount`, `disconnectCount`, `packetsTotal`, `packetsLast5m`
(sliding 5-min window via per-second buckets keyed by unix second — no
stale-leak), `lastError`. Wired at the existing OnConnect /
ConnectionLost / DefaultPublish callsites alongside the liveness
watchdog. Idempotent registration so counters survive reconnects.
Snapshot emitted in the existing stats file under `source_statuses`
(additive, `omitempty`).
- **Backend:** new `GET /api/mqtt/status` handler reads the ingestor
stats file and returns the per-source list. **Broker passwords are
masked** via a regex over the `scheme://user:pass@host` form (covers
mqtt/mqtts/tcp/ssl/ws/wss). Mask is also applied to `lastError` as
defense-in-depth (broker libs occasionally quote the failing URL).
OpenAPI completeness gate satisfied with a `routeDescriptions` entry.
- **Frontend:** small self-contained panel
(`public/mqtt-status-panel.js`) mounted above the Observers table.
Auto-refreshes every 10s, color-codes each row (green = connected +
recent packet, yellow = connected idle, red = disconnected), and tears
down its timer on SPA route change.
## TDD
- Red commit `f19a93b5` — stub `/api/mqtt/status` handler + assertion
test that the broker password is `****`-redacted. Test fails on the
assertion (handler passes the URL through verbatim). Compile-clean —
assertion-fail, not build-fail.
- Green commit `77042e41` — `maskBrokerURL` helper + table-driven unit
tests across all schemes + handler rewires to mask both `Broker` and
`LastError`.
- Subsequent commits land the ingestor wiring and the frontend panel.
## Tests
```
$ cd cmd/server && go test -run 'TestMqttStatus|TestMaskBrokerURL' -v ./...
PASS: TestMqttStatus_MasksBrokerPassword
PASS: TestMqttStatus_EmptyWhenNoStatsFile
PASS: TestMaskBrokerURL_Patterns (10 subtests)
$ cd cmd/ingestor && go test -run 'TestSourceStatus|TestSnapshotSourceStatuses' -v ./...
PASS: TestSourceStatus_BasicLifecycle
PASS: TestSourceStatus_Disconnect
PASS: TestSnapshotSourceStatuses_ReturnsAll
$ node test-mqtt-status-panel.js
7 passed, 0 failed
```
Full `go test ./...` clean in both `cmd/server` and `cmd/ingestor`.
## Preflight overrides
- `cross-stack`: justified — issue #1043 is intrinsically full-stack
(ingestor stats → server endpoint → observers panel). Per-stack split
would land an unreachable endpoint or a fetch with no backend.
- `check-xss-sinks` (public/mqtt-status-panel.js:55): justified — the
flagged `innerHTML=` is a fully-static literal (empty-state placeholder,
no payload data interpolated). All payload-bearing `innerHTML=` sites in
this file run through `escapeHTML` (defined in the same file); the test
`renderPanel never echoes a plaintext password (defense-in-depth)`
exercises the rendered HTML against payload strings.
## Acceptance criteria
- [x] `/api/mqtt/status` returns per-source connection state —
`cmd/server/mqtt_status.go`
- [x] UI panel shows all configured sources with live status —
`public/mqtt-status-panel.js`
- [x] Connection state updates on reconnect/disconnect events —
`MarkConnect` / `MarkDisconnect` wired in `cmd/ingestor/main.go`
- [x] Broker URLs don't expose passwords in the API response —
`maskBrokerURL` + 13 test cases
- [x] Works with 1-N sources — registry is keyed per-source, snapshot
iterates the map
**Partial fix for #1043** — per-packet `mqtt_source` attribution (the
issue's "Follow-up" section) is **deferred** per the `mc-bot-triaged:v1`
triage and the autofix comment ("Per-packet attribution deferred to
follow-up issue"). That work requires a new observation-row column and
DB schema migration, both explicitly out of scope for this PR.
Refs #1043
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
**Red commit:** f6290b63 — CI run will appear at
https://github.com/Kpa-clawbot/CoreScope/actionsFixes#1283.
## What
Moves all four DB write operations out of `cmd/server/` into
`cmd/ingestor/`, making the server truly read-only and eliminating the
SQLITE_BUSY VACUUM bug at its root: the server can no longer race the
ingestor for the write lock because the server has no write path.
## The four operations
| # | Was in | Now in |
|---|--------|--------|
| 1 | `cmd/server/vacuum.go` (`checkAutoVacuum`, full VACUUM +
`auto_vacuum=INCREMENTAL` migration) | `cmd/ingestor/db.go`
`Store.CheckAutoVacuum` (already existed; ingestor runs it at startup
**before** the MQTT subscriber starts → no contention) |
| 2 | `cmd/server/db.go` `PruneOldPackets` (`DELETE FROM transmissions`)
| `cmd/ingestor/maintenance.go` `Store.PruneOldPackets` (new) + 24h
ticker in `cmd/ingestor/main.go` |
| 3 | `cmd/server/db.go` `PruneOldMetrics` (`DELETE FROM
observer_metrics`) | `cmd/ingestor/db.go` `Store.PruneOldMetrics`
(already existed) |
| 4 | `cmd/server/db.go` `RemoveStaleObservers` (`UPDATE observers SET
inactive=1`) | `cmd/ingestor/db.go` `Store.RemoveStaleObservers`
(already existed) |
## HTTP surface
- **Removed:** `POST /api/admin/prune` (`handleAdminPrune`, route,
openapi entry). Operators trigger an ad-hoc prune by restarting the
ingestor.
- **Kept:** `GET /api/backup` — uses `VACUUM INTO` which writes to a
separate file, not the live DB; read-only-safe.
## Tests
- `cmd/server/readonly_invariant_test.go` (RED gate) — reflect-asserts
`PruneOldPackets`/`PruneOldMetrics`/`RemoveStaleObservers` are NOT
methods on the server's `*DB`. Fails on master, passes after this PR.
- `cmd/ingestor/issue1283_test.go` — exercises `Store.PruneOldPackets`
and the auto_vacuum=NONE → INCREMENTAL migration through
`Store.CheckAutoVacuum` with `vacuumOnStartup=true`.
## Why the bug is gone
The SQLITE_BUSY VACUUM failure happened because supervisord launched
both ingestor + server in one container; the ingestor took the write
lock for INSERTs and the server's `checkAutoVacuum` then failed to
acquire it within `busy_timeout=5000`. After this PR, only the ingestor
ever opens a writable connection, and it runs `CheckAutoVacuum`
**before** spawning the MQTT subscriber → no contention possible.
## Scope notes
- `cachedRW()` still has three pre-existing callers in `cmd/server/`
(`neighbor_persist.go`, `ensure_indexes.go`,
`from_pubkey_migration.go`). These pre-date #1283 and are not in the
issue's four-operation list. Leaving them for follow-up keeps this PR
honest about scope; AGENTS.md documents the invariant so new write paths
can't sneak in.
- PII preflight reports false positives on the Go method name
`requireAPIKey` in `routes.go` diff context — no real PII.
- Server-side neighbor-edge prune (`PruneNeighborEdges`) intentionally
left in place — out of scope of #1283.
---------
Co-authored-by: MeshCore Bot <bot@meshcore.local>
## Summary
Implements `GET /api/backup` — one-click SQLite database export per
#474.
Operators can now grab a complete, consistent snapshot of the analyzer
DB with a single authenticated request — no SSH, no scripts, no DB
tooling.
## Endpoint
```
GET /api/backup
X-API-Key: <key> # required
→ 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="corescope-backup-<unix>.db"
<body: complete SQLite database file>
```
## Approach
Uses SQLite's `VACUUM INTO 'path'` to produce an atomic, defragmented
copy of the database into a fresh file:
- **Consistent**: VACUUM INTO runs at read isolation — the snapshot
reflects a single point in time even while the ingestor is writing to
the WAL.
- **Non-blocking**: writers continue uninterrupted; we never hold a
write lock.
- **Works on read-only connections**: verified manually against a
WAL-mode source DB (`mode=ro` connection successfully produces a
snapshot).
- **No corruption risk**: even if the live on-disk DB has issues, VACUUM
INTO surfaces what the server can read rather than copying broken pages
byte-for-byte.
The snapshot is staged in `os.MkdirTemp(...)` and removed after the
response body is fully streamed (deferred cleanup). Requesting client IP
is logged for audit.
The issue suggested an alternative in-memory rebuild path; `VACUUM INTO`
is simpler, faster, and produces a strictly more accurate copy of what
the server actually sees, so going with it.
## Security
- Mounted under `requireAPIKey` middleware — same gate as other admin
endpoints (`/api/admin/prune`, `/api/perf/reset`).
- Returns 401 without a valid `X-API-Key` header.
- Returns 403 if no API key is configured server-side.
- `X-Content-Type-Options: nosniff` set on the response.
## TDD
- **Red** (`99548f2`): `cmd/server/backup_test.go` adds
`TestBackupRequiresAPIKey` + `TestBackupReturnsValidSQLiteSnapshot`.
Stub handler returns 200 with no body so the tests fail on assertions
(Content-Type / Content-Disposition / SQLite magic header), not on
import or build errors.
- **Green** (`837b2fe`): real implementation lands; both tests pass;
full `go test ./...` suite stays green.
## Files
- `cmd/server/backup.go` — handler implementation
- `cmd/server/backup_test.go` — red-then-green tests
- `cmd/server/routes.go` — route registration under `requireAPIKey`
- `cmd/server/openapi.go` — OpenAPI metadata so `/api/openapi`
advertises the endpoint
## Out of scope (follow-ups)
- Rate limiting (issue suggested 1 req/min). Not added here —
admin-key-gated endpoint with a fast snapshot path is acceptable for v1;
happy to add a token-bucket limiter in a follow-up if operators report
hammering.
- UI button to trigger the download (frontend work — separate PR).
Fixes#474
---------
Co-authored-by: corescope-bot <bot@corescope.local>
## Summary
Auto-generated OpenAPI 3.0.3 spec endpoint (`/api/spec`) and Swagger UI
(`/api/docs`) for the CoreScope API.
## What
- **`cmd/server/openapi.go`** — Route metadata map
(`routeDescriptions()`) + spec builder that walks the mux router to
generate a complete OpenAPI 3.0.3 spec at runtime. Includes:
- All 47 API endpoints grouped by tag (admin, analytics, channels,
config, nodes, observers, packets)
- Query parameter documentation for key endpoints (packets, nodes,
search, resolve-hops)
- Path parameter extraction from mux `{name}` patterns
- `ApiKeyAuth` security scheme for API-key-protected endpoints
- Swagger UI served as a self-contained HTML page using unpkg CDN
- **`cmd/server/openapi_test.go`** — Tests for spec endpoint (validates
JSON structure, required fields, path count, security schemes,
self-exclusion of `/api/spec` and `/api/docs`), Swagger UI endpoint, and
`extractPathParams` helper.
- **`cmd/server/routes.go`** — Stores router reference on `Server`
struct for spec generation; registers `/api/spec` and `/api/docs`
routes.
## Design Decisions
- **Runtime spec generation** vs static YAML: The spec walks the actual
router, so it can never drift from registered routes. Route metadata
(summaries, descriptions, tags, auth flags) is maintained in a parallel
map — the test enforces minimum path count to catch drift.
- **No external dependencies**: Uses only stdlib + existing gorilla/mux.
Swagger UI loaded from unpkg CDN (no vendored assets).
- **Security tagging**: Auth-protected endpoints (those behind
`requireAPIKey` middleware) are tagged with `security: [{ApiKeyAuth:
[]}]` in the spec, matching the actual middleware configuration.
## Testing
- `go test -run TestOpenAPI` — validates spec structure, field presence,
path count ≥ 20, security schemes
- `go test -run TestSwagger` — validates HTML response with swagger-ui
references
- `go test -run TestExtractPathParams` — unit tests for path parameter
extraction
---------
Co-authored-by: you <you@example.com>
dborup