Follow-up to #1806, which decoded the raw fields but left them
unfriendly to read:
- Detail sidepanel no longer falls through to a generic "Raw" row for
CONTROL packets -- proper Subtype/Filter/Tag/Since/SNR/Pubkey rows
with byte offsets, matching the ADVERT/TRACE field-table pattern.
- DISCOVER_REQ's filter byte is a per-type bitmask (firmware checks
`filter & (1 << ADV_TYPE_x)`); now rendered as type names
(Companion/Repeater/Room Server/Sensor) instead of raw hex.
- DISCOVER_RESP's node_type nibble gets the same human-readable label
(shares the ADV_TYPE_* enum already used for ADVERT roles).
- SNR converted from the wire int8 (value*4) to real dB, matching the
/4.0 conversion TRACE's snrValues already apply in both decoders.
- Responder pubkey (full 32B or 8B prefix) resolved to a clickable
node link in the detail panel via /api/nodes/{pubkey}, which already
handles prefix resolution server-side (#772); falls back to the
first 8 hex chars when unresolved. The packet-row preview (rendered
per-row, no live lookup) always shows the 8-hex-char form.
Frontend-only (public/packets.js) -- the ingestor's decodeControl()
already emits every field used here (#1806); this just makes the
existing data readable.
Ranks configured areas by number of distinct pings with a relay hop or
hearing station inside them -- "which area is most active" -- reusing
the same touched-areas config/matching View Path already draws from
(annotatePacketPathTouchedAreas), rather than reimplementing area
matching. Omitted entirely on deployments with no areas configured.
The all-time records lock in forever once someone sets a big one (e.g.
a 364km farthest ping) -- ThisWeek mirrors the same 5 slots on a
rolling 7-day window so there's an achievable target to chase
regularly, shown alongside (not instead of) the all-time board.
Unlike the all-time records and the Top Relays/Top First-Hearers boards,
"who pings the most" is more meaningful as an ongoing thing than a
fixed leaderboard someone can win once and hold forever.
dborup asked for suggestions on what else to add to Ping Scores; picked
"Top Senders" from the shortlist -- the page had Top Relays and Top
First-Hearers but nothing ranking who actually sends the most pings.
Backend: new SenderLeaderboard field on PingScoresSnapshot, tallied
alongside the existing relay/observer leaderboards in
computeAllPingScores. Keyed by the sender display name straight from
the channel message (ping_triggers.sender) -- unlike relays/observers
there's no resolved pubkey to link back to a node, so entries
deliberately never carry one (PingLeaderboardEntry.Pubkey stays
omitted).
Frontend: leaderboardTableHtml gained an optional column-label param
("Sender" instead of the default "Node") since sender entries aren't
resolvable mesh nodes and "Node" would be misleading there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: clicking the link just landed on a bare "#/map" with no way to
share it -- state lived in sessionStorage, gone after the click and
never in the URL to begin with.
Switched to a deep link, #/map?estimatedNodes=1, same convention as the
existing packet-path deep link (#/map?packet=<hash>&obs=<id>,
loadRouteFromDeepLink). map.js's new loadEstimatedNodesFromDeepLink
fetches /api/analytics/areas fresh and draws the result -- no
sessionStorage involved, since this list isn't per-request state, just
the deployment's current estimate, so the URL alone reproduces it.
Copy/share/reload all just work now. Also converted the trigger from a
JS-click <button> to a real <a href> so "Copy Link Address" works
without even clicking through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: the "Filter by observer or neighbor name..." placeholder didn't
fit inside the 320px max-width box. Widened to 420px (width:100% so it
still shrinks on narrow viewports) and trimmed the placeholder slightly
for margin.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: "kan vi have en panel med scopes vi ikke kender på corescope
som observer neighbors har fundet" -- a panel showing region-scope
names that turn up in neighbors' reported OTA scope lists but aren't
part of this deployment's configured hashRegions.
Backend: computeUnknownScopes (db.go) is a pure function over the same
AllObserverNeighborsEntry rows /api/observers/neighbors already fetches
-- no second query. Parses each neighbor's comma-joined scopes string,
excludes the "*" wildcard, normalizes missing "#" prefixes, diffs
against regionutil.NormalizeNames(cfg.HashRegions), and counts DISTINCT
neighbors per unknown scope (not raw rows, which would double-count a
neighbor seen by multiple observers) with up to 5 example neighbor
names. New unknownScopes field on the existing response.
Frontend: new "Scopes CoreScope Doesn't Know About Yet" panel on the
Observer Neighbors tool page, rendered above the main table, empty
(nothing shown) when there's nothing unknown -- absence is the normal
case, not an empty-state message.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Caught live on stg: clicking a column sorted correctly (confirmed via
sortValue/data ordering) but no header ever visually highlighted as the
active sort column -- the thead template hardcoded class="sortable" on
every <th> and never added sort-active, even though sortArrow() computed
the right up/down glyph. Refactored into a sortTh(col, label) helper so
the class and arrow can't drift apart again; added a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup wanted a single place to see all direct neighbors observed
across every observer, rather than clicking into each one individually.
Discussed placement -- landed on Tools, matching the existing Path
Inspector/Trace Viewer pattern.
Backend: GetAllObserverNeighbors flattens every observer_neighbors row
network-wide, joined with observer/neighbor display names, cross-referenced
against the packet-derived neighbor_edges graph exactly like the
per-observer endpoint (memoized per observer_id to avoid redundant
queries). Reads the full result set into memory before running any
packetGraphNeighbors lookups -- issuing a query per row while the main
cursor is still open is the single-connection-pool deadlock class from
earlier this session. New GET /api/observers/neighbors, registered
before /api/observers/{id} since both are 3-segment patterns and
gorilla/mux matches by registration order. Blacklisted observers excluded.
Frontend: new Tools > Observer Neighbors page (observer-neighbors-tool.js),
sortable by any column + a client-side observer/neighbor name filter,
reusing the col-scope-list wrap fix from the per-observer panel and the
same row semantics (unresolved pubkey truncated/unlinked, "no reply" for
scope-query timeouts, packet-evidence confirmed/not-seen-yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: a catch-all/multi-region observer's scope list (e.g.
"*,#eu,#dk,#dk-sjl,#nordic,...") was unreadable in the Direct Neighbors
panel. The generic .data-table td rule (max-width:0 + ellipsis, tuned
for short cells) was clipping it almost entirely. New col-scope-list
override lets that column wrap instead, same pattern as the existing
col-details override.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup asked for a map view of the estimated-position nodes, like View
Path's approximate markers off the Pong reply. Backend now returns
estimatedNodes -- the flat, network-wide list behind Position-Fix
Coverage Gaps' Approximated counts, with each node's actual estimated
lat/lon (computeAreaPositionGaps previously only tallied per-area
counts and discarded the coordinates nearestPositionedNeighbor already
computed). A new button on the Areas tab stores that list in
sessionStorage and opens the Map page, where a new drawEstimatedNodes
renderer (modeled on drawGPSTrail) plots each as a dashed, semi-transparent
circleMarker -- same "this position is a guess" visual language as View
Path's approx markers, sized/opacity-scaled by contributor count.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup asked to be able to sort alphabetically, by highest value, etc.
in any column, not just the built-in default ordering. Adds click-to-sort
headers (same sortable/sort-active/sort-arrow convention as the Channels
table) to Density, Bridge Nodes, and Position Gaps, refactoring the
collapse-to-10 helper into a combined collapsible+sortable
makeAreasSection controller. Free-text columns (Role Mix, Which Areas)
stay unsortable -- no single meaningful value to sort a list/breakdown
by. Density's pre-click default (worst-health-first) is a composite of
Degraded+Silent/Total, so no single header starts highlighted; Bridge
Nodes and Position Gaps default to a real column (otherAreaCount /
pctEstimated desc) that does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: numbers didn't line up under their column headers across all
three Areas tab tables (Density, Bridge Nodes, Position Gaps) -- td
cells had text-align:right but the matching th headers didn't, so
header text sat flush-left while the numbers under it sat flush-right.
Added a regression test asserting right-aligned header count matches
right-aligned cell count per table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup asked to apply the same treatment given to Position-Fix Coverage
Gaps to the other two Areas sections. Node Density & Health now sorts
worst-health-first (% degraded+silent descending) instead of the API's
raw Total-desc order, so small struggling areas surface instead of
being buried under Europa/Danmark. Cross-Area Bridge Nodes keeps its
existing otherAreaCount-desc order (already most-interesting-first) and
just gets the same collapse-to-10 treatment. Factored the three
sections' collapsible-table rendering/wiring into shared helpers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup: the full 36-row dump buried the handful of areas with an actual
gap behind dozens already fully GPS-mapped (0% estimated). Sort by %
estimated descending and collapse to top 10 with a "Show all" toggle,
matching the existing pattern from the Wardriving tab's Top Senders /
Coverage by Observer sections.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New GET /api/analytics/areas endpoint and "Areas" analytics tab, built on
the configured drawn-polygon Areas (meshguide.dk sync), distinct from
hashRegion scope adoption:
- Density: node count + active/degraded/silent health + role mix per
area, multi-membership via AreaKeysForPoint (a node in a sub-area also
rolls up into its parent region).
- Bridge nodes: which nodes' packet-derived neighbor_edges reach into
another area, ranked by how many other areas they touch -- distinct
from the network-wide, area-unaware bridge_score.
- Position gaps: per area, real GPS fix vs. neighbor-centroid-estimated
position, reusing nearestPositionedNeighbor (the same technique View
Path's approx markers use) purely as an internal coverage signal.
30s TTL cache on the handler since position-gap computation calls
nearestPositionedNeighbor once per unpositioned node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SaarMesh-Bot flagged on PR #1867 that the server's one-time detectSchema()
PRAGMA snapshot racing the ingestor's ALTER TABLE migrations isn't specific
to configured_scope -- it's a general property of every optional column,
and offered to send a fix. Built it ourselves instead.
hasResolvedPath/hasObsRawHex/hasScopeName/hasDefaultScope/
hasConfiguredScope/hasMultibyteSupCols/hasLastSeen are now schemaFlag
(atomic.Bool) methods instead of plain bools, self-healed by a background
ticker (healSchemaFlags) started once in OpenDB and stopped in Close().
Unlike an eager reprobe-on-read, the healer never nests inside another
caller's already-open *sql.Rows cursor, so it can't self-deadlock a
single-connection pool the way a naive "reprobe inside get()" version of
this fix did (caught immediately by the existing test suite hanging).
detectSchemaWithRetry's fixed 150ms budget is gone -- replaced by an
unbounded self-heal that catches the migration whenever it actually lands.
Every read site across db.go/store.go/chunked_load.go/main.go is now a
method call; test fixtures that previously force-set the old bool fields
now call .forceTrue() on the underlying schemaFlag.
borderColor was set to raw 'var(--accent)'/'var(--status-yellow)'
strings, which Canvas 2D (unlike SVG/DOM) can't resolve as a strokeStyle
-- Chart.js silently fell back to its default black for both lines,
making the SNR and Heard(s ago) series indistinguishable (spotted live
on stg). Switched to this page's existing CHART_COLORS literal-hex
palette, matching the pattern the other 4 charts on this page already
use. The small inline SVG sparkline was unaffected -- SVG is real DOM
and does resolve CSS custom properties, just Canvas doesn't.
dborup: "det ville være cool wih more history... jeg vil bare gerne se
mere" -- not more retention, more detail in the existing view.
- neighborSnrSparkline now labels min/max dB directly on the sparkline,
readable without hovering.
- Clicking a sparkline (any row with data) opens a modal with a bigger
Chart.js line chart showing both SNR and heard_secs_ago on dual y-axes,
with native hover tooltips. Reuses the existing generic .modal-overlay/
.modal CSS (previously only exercised by the channel-add modal).
- Metrics are cached per-pubkey when the sparkline loads, so opening the
modal needs no second fetch.
- Click delegation is wired once at module load (not per-render), since
renderDetail rewrites #obsDetailContent's innerHTML on every load.
dborup spotted the raw /neighbors payload also carries snr and
heard_secs_ago per neighbor -- previously dropped entirely by
handleNeighborsReport's parsing loop. Both fields are present regardless
of scope-query status (responded or timeout).
New append-only observer_neighbor_metrics table, inspired by the existing
RF Health tab's observer_metrics pattern: one row per neighbor per
/neighbors report, pruned on the same 30-day MetricsRetentionDays
schedule. Deliberately no ordering guard on the write side (unlike the
current-only observer_neighbors snapshot) -- every report is valid
history at its own timestamp.
GET /api/observers/{id}/neighbors/{pubkey}/metrics serves the raw
(undownsampled) history. Frontend renders a per-row SNR sparkline on the
Direct Neighbors panel using the same lightweight inline-SVG technique as
the RF Health grid's noise-floor sparklines (no Chart.js), loaded async
after the panel paints.
dborup's suggested "more ambitious use": flag firmware-confirmed neighbors
that our packet-path-inferred neighbor_edges graph has never seen adjacent
-- a diagnostic for coverage gaps / packet loss, not itself a fault.
GetObserverNeighbors now annotates each entry with seenViaPackets, checked
against neighbor_edges in both column directions (canonEdge orders
node_a<=node_b). Surfaced as a neutral "confirmed"/"not seen yet" column
on the Direct Neighbors panel, explaining the two possible causes in the
tooltip. This surfaces the mismatch only -- it does not yet feed the
path-hop disambiguator's own candidate scoring, which would be a separate,
larger change.
Surfaces the observer's own firmware-reported zero-hop neighbor set --
ground truth from /neighbors, distinct from the packet-path-inferred
neighbor_edges graph. Requested by dborup after shipping the last-report
tracking: "should we show what neighbors it has under observer stat".
New observer_neighbors table (current-only snapshot, full replace on every
report so a dropped neighbor disappears rather than lingering). Guarded
against out-of-order reports by checking the report timestamp against the
already-touched observers.last_neighbors_report_at before replacing.
GET /api/observers/{id}/neighbors joins against nodes for name/role;
absence (never reported, or an unresolved pubkey) always renders neutrally
-- empty array not null, no warning icon -- consistent with the last
feature's no-shame requirement.
The Observer DB struct carried the field correctly, but handleObservers/
handleObserverDetail build a separate field-by-field ObserverResp DTO for
the actual JSON response, which never picked it up -- caught live on stg
(field missing from /api/observers entirely). Added handler-level tests
that decode the JSON response, since the existing DB-layer test for this
class of field didn't reach the DTO conversion at all.
cwichura asked on PR #1867 for a way to identify which observers have the
opt-in /neighbors firmware feature enabled, to help contact operators about
enabling it -- explicitly asking that the UI not "shame" observers that
lack it (non-PSRAM hardware can't send it; other MQTT uploaders haven't
been updated to include it).
Adds observers.last_neighbors_report_at, touched by the ingestor's new
TouchObserverNeighborsReport once per /neighbors MQTT message regardless
of whether any neighbor carried scope evidence. Added to dbschema's
AssertReady mustCol list from day one so the server can read it
unconditionally with no PRAGMA-detection race (#1321 pattern).
Surfaced as a sortable "Neighbors" column on the observers list and a
stat card on the observer detail page -- both render a neutral dash/
"never" with an explanatory tooltip when absent, never a warning icon.
SaarMesh-Bot flagged on #1865 (PR #1867 commit 631686a) that
UpdateNodeConfiguredScope stored the report timestamp raw and compared it
lexicographically. Real firmware emits microsecond+offset timestamps and
different observers may use "Z" or non-UTC offsets, so string comparison
can diverge from chronological order and let a stale report win.
normalizeReportTS parses RFC3339Nano/RFC3339 and stores canonical UTC
RFC3339, used for both storage and the comparison.
cwichura flagged on PR #1867 that nodes.configured_scope (from the
/neighbors OTA report) was stored as the observer's raw scope string
("dk") while default_scope and every other scope display use the
#-prefixed hashRegion form ("#dk"). Apply the same regions.Normalize
already used for default_scope, with "*" passed through unprefixed
since it's a protocol wildcard, not a region name.
Two dborup-specific, already-used pieces held back from the generic
master branch:
- ops/meshguide-sync/sync_areas.py: fetches Danish region boundaries
from meshguide.dk, useless for any other CoreScope deployment. This
fast-forward merge brought it in again (areas-meshguide-sync had it
restored after an earlier accidental deletion); excluding it here,
same as every previous merge round.
- cmd/ingestor/db.go's ping_triggers_backfill_v1 async migration: a
one-time historical scan that already ran and completed on stg
(298 pings recorded from before the feature existed). Per dborup:
other deployments merging from master didn't ask for a full-table
CHAN-message scan on their own database. The ping-scores feature
itself (detection, scoring, leaderboards, frontend) stays -- only
the backfill-specific function + its tests are removed.
Both pieces remain intact on areas-meshguide-sync, where they've
already served their purpose.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dborup