dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-06-05 21:41:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,285 Commits 89 Branches 30 Tags
fe997fefb2b083062ee252f957b147cd6a3de106
Commit Graph

7 Commits

Author SHA1 Message Date
Kpa-clawbot 7179afcfde feat(#1228): reject geo-implausible neighbor-graph edges at build time (#1230) 
Fixes #1228 — geo-implausible neighbor-graph edges are rejected at build
time.

Red commit: `5a6d9660` — failing tests for 4 cases (reject SF↔Berlin,
accept local CA, accept no-GPS endpoint, counter increments). Live CI
run (latest commit):
https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1228

## Why

The disambiguator's tier-1 affinity graph is built blindly from path
co-occurrence. On wide-geo MQTT deployments, a single bad hop
disambiguation seeds an edge across geographically impossible distances
(e.g. Bay Area ↔ Berlin), which then reinforces the same wrong
resolution next time. Self-poisoning spiral.

## What changed

- `upsertEdge` now consults a per-graph GPS index. When **both**
endpoints have known GPS and their haversine distance exceeds the
threshold, the edge is dropped and `NeighborGraph.RejectedEdgesGeoFar`
(atomic) is incremented.
- Either endpoint missing GPS ⇒ accept (no signal to reject), per
acceptance criteria.
- Threshold is configurable via `neighborGraph.maxEdgeKm` (default **500
km** — well above any plausible terrestrial LoRa hop, including
satellite-assisted). 0 ⇒ use default; negative ⇒ disable the filter.
Exposed via `Config.NeighborMaxEdgeKm()`.
- New `BuildFromStoreWithOptions` carrying the threshold;
`BuildFromStore` and `BuildFromStoreWithLog` are kept as thin wrappers.
- Stats are surfaced under `GET /api/analytics/neighbor-graph` as
`stats.rejected_edges_geo_far`.
- All rejection logs PII-truncate pubkeys to 8 hex chars (public repo
discipline).
- `config.example.json` updated with the new field + comment.

## Follow-up

#1229 (per-region scoped affinity graphs) depends on this landing first.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
 2026-05-16 10:14:44 -07:00
Joel Claw b9ba447046 feat: add nodeBlacklist config to hide abusive/troll nodes (#742) 
## Problem

Some mesh participants set offensive names, report deliberately false
GPS positions, or otherwise troll the network. Instance operators
currently have no way to hide these nodes from public-facing APIs
without deleting the underlying data.

## Solution

Add a `nodeBlacklist` array to `config.json` containing public keys of
nodes to exclude from all API responses.

### Blacklisted nodes are filtered from:

- `GET /api/nodes` — list endpoint
- `GET /api/nodes/search` — search results
- `GET /api/nodes/{pubkey}` — detail (returns 404)
- `GET /api/nodes/{pubkey}/health` — returns 404
- `GET /api/nodes/{pubkey}/paths` — returns 404
- `GET /api/nodes/{pubkey}/analytics` — returns 404
- `GET /api/nodes/{pubkey}/neighbors` — returns 404
- `GET /api/nodes/bulk-health` — filtered from results

### Config example

```json
{
  "nodeBlacklist": [
    "aabbccdd...",
    "11223344..."
  ]
}
```

### Design decisions

- **Case-insensitive** — public keys normalized to lowercase
- **Whitespace trimming** — leading/trailing whitespace handled
- **Empty entries ignored** — `""` or `" "` do not cause false positives
- **Nil-safe** — `IsBlacklisted()` on nil Config returns false
- **Backward-compatible** — empty/missing `nodeBlacklist` has zero
effect
- **Lazy-cached set** — blacklist converted to `map[string]bool` on
first lookup

### What this does NOT do (intentionally)

- Does **not** delete or modify database data — only filters API
responses
- Does **not** block packet ingestion — data still flows for analytics
- Does **not** filter `/api/packets` — only node-facing endpoints are
affected

## Testing

- Unit tests for `Config.IsBlacklisted()` (case sensitivity, whitespace,
empty entries, nil config)
- Integration tests for `/api/nodes`, `/api/nodes/{pubkey}`,
`/api/nodes/search`
- Full test suite passes with no regressions
 2026-04-17 23:43:05 +00:00
Kpa-clawbot ceea136e97 feat: observer graph representation (M1+M2) (#774) 
## Summary

Fixes #753 — Milestones M1 and M2: Observer nodes in the neighbor graph
are now correctly labeled, colored, and filterable.

### M1: Label + color observers

**Backend** (`cmd/server/neighbor_api.go`):
- `buildNodeInfoMap()` now queries the `observers` table after building
from `nodes`
- Observer-only pubkeys (not already in the map as repeaters etc.) get
`role: "observer"` and their name from the observers table
- Observer-repeaters keep their repeater role (not overwritten)

**Frontend**:
- CSS variable `--role-observer: #8b5cf6` added to `:root`
- `ROLE_COLORS.observer` was already defined in `roles.js`

### M2: Observer filter checkbox (default unchecked)

**Frontend** (`public/analytics.js`):
- Observer checkbox added to the role filter section, **unchecked by
default**
- Observers create hub-and-spoke patterns (one observer can have 100+
edges) that drown out the actual repeater topology — hiding them by
default keeps the graph clean
- Fixed `applyNGFilters()` which previously always showed observers
regardless of checkbox state

### Tests

- Backend: `TestBuildNodeInfoMap_ObserverEnrichment` — verifies
observer-only pubkeys get name+role from observers table, and
observer-repeaters keep their repeater role
- All existing Go tests pass
- All frontend helper tests pass (544/544)

---------

Co-authored-by: you <you@example.com>
 2026-04-16 21:35:14 -07:00
Kpa-clawbot 9917d50622 fix: resolve neighbor graph duplicate entries from different prefix lengths (#699) 
## Problem

The neighbor graph creates separate entries for the same physical node
when observed with different prefix lengths. For example, a 1-byte
prefix `B0` (ambiguous, unresolved) and a 2-byte prefix `B05B` (resolved
to Busbee) appear as two separate neighbors of the same node.

Fixes #698

## Solution

### Part 1: Post-build resolution pass (Phase 1.5)

New function `resolveAmbiguousEdges(pm, graph)` in `neighbor_graph.go`:
- Called after `BuildFromStore()` completes the full graph, before any
API use
- Iterates all ambiguous edges and attempts resolution via
`resolveWithContext` with full graph context
- Only accepts high-confidence resolutions (`neighbor_affinity`,
`geo_proximity`, `unique_prefix`) — rejects
`first_match`/`gps_preference` fallbacks to avoid false positives
- Merges with existing resolved edges (count accumulation, max LastSeen)
or updates in-place
- Phase 1 edge collection loop is **unchanged**

### Part 2: API-layer dedup (defense-in-depth)

New function `dedupPrefixEntries()` in `neighbor_api.go`:
- Scans neighbor response for unresolved prefix entries matching
resolved pubkey entries
- Merges counts, timestamps, and observers; removes the unresolved entry
- O(n²) on ~50 neighbors per node — negligible cost

### Performance

Phase 1.5 runs O(ambiguous_edges × candidates). Per Carmack's analysis:
~50ms at 2K nodes on the 5-min rebuild cycle. Hot ingest path untouched.

## Tests

9 new tests in `neighbor_dedup_test.go`:

1. **Geo proximity resolution** — ambiguous edge resolved when candidate
has GPS near context node
2. **Merge with existing** — ambiguous edge merged into existing
resolved edge (count accumulation)
3. **No-match preservation** — ambiguous edge left as-is when prefix has
no candidates
4. **API dedup** — unresolved prefix merged with resolved pubkey in
response
5. **Integration** — node with both 1-byte and 2-byte prefix
observations shows single neighbor entry
6. **Phase 1 regression** — non-ambiguous edge collection unchanged
7. **LastSeen preservation** — merge keeps higher LastSeen timestamp
8. **No-match dedup** — API dedup doesn't merge non-matching prefixes
9. **Benchmark** — Phase 1.5 with 500+ edges

All existing tests pass (server + ingestor).

---------

Co-authored-by: you <you@example.com>
 2026-04-10 11:19:54 -07:00
efiten b587f20d1c feat: add distance column to neighbor table in node details (#617) 
Closes #616

## What

Adds a **Distance** column to the neighbor table on the node detail
page.

When both the viewed node and a neighbor have GPS coordinates recorded,
the table shows the haversine distance between them (e.g. `3.2 km`).
When either node lacks GPS, the cell shows `—`.

## Changes

**Backend** (`cmd/server/neighbor_api.go`):
- Added `distance_km *float64` (omitempty) to `NeighborEntry`
- In `handleNodeNeighbors`: look up source node coords from `nodeMap`,
then for each resolved (non-ambiguous) neighbor with GPS, compute
`haversineKm` and set the field

**Frontend** (`public/nodes.js`):
- Added `Distance` column header between Last Seen and Conf
- Cell renders `X.X km` or `—` (muted) when unavailable

**Tests** (`cmd/server/neighbor_api_test.go`):
- `TestNeighborAPI_DistanceKm_WithGPS`: two nodes with real coords →
`distance_km` is positive
- `TestNeighborAPI_DistanceKm_NoGPS`: two nodes at 0,0 → `distance_km`
is nil

## Verification

Test at **https://staging.on8ar.eu** — navigate to any node detail page
and scroll to the Neighbors section. Nodes with GPS coordinates show a
distance; those without show `—`.
 2026-04-05 12:33:23 -07:00
Kpa-clawbot 58f791266d feat: affinity debugging tools (#482) — milestone 6 (#521) 
## Summary

Milestone 6 of #482: Observability & Debugging tools for the neighbor
affinity system.

These tools exist because someone will need them at 3 AM when "Show
Neighbors is showing the wrong node for C0DE" and they have 5 minutes to
diagnose it.

## Changes

### 1. Debug API — `GET /api/debug/affinity`
- Full graph state dump: all edges with weights, observation counts,
last-seen timestamps
- Per-prefix resolution log with disambiguation reasoning (Jaccard
scores, ratios, thresholds)
- Query params: `?prefix=C0DE` filter to specific prefix,
`?node=<pubkey>` for specific node's edges
- Protected by API key (same auth as `/api/admin/prune`)
- Response includes: edge count, node count, cache age, last rebuild
time

### 2. Debug Overlay on Map
- Toggle-able checkbox "🔍 Affinity Debug" in map controls
- Draws lines between nodes showing affinity edges with color coding:
  - Green = high confidence (score ≥ 0.6)
  - Yellow = medium (0.3–0.6)
  - Red = ambiguous (< 0.3)
- Line thickness proportional to weight, dashed for ambiguous
- Unresolved prefixes shown as ❓ markers
- Click edge → popup with observation count, last seen, score, observers
- Hidden behind `debugAffinity` config flag or
`localStorage.setItem('meshcore-affinity-debug', 'true')`

### 3. Per-Node Debug Panel
- Expandable "🔍 Affinity Debug" section in node detail page (collapsed
by default)
- Shows: neighbor edges table with scores, prefix resolutions with
reasoning trace
- Candidates table with Jaccard scores, highlighting the chosen
candidate
- Graph-level stats summary

### 4. Server-Side Structured Logging
- Integrated into `disambiguate()` — logs every resolution decision
during graph build
- Format: `[affinity] resolve C0DE: c0dedad4 score=47 Jaccard=0.82 vs
c0dedad9 score=3 Jaccard=0.11 → neighbor_affinity (ratio 15.7×)`
- Logs ambiguous decisions: `scores too close (12 vs 9, ratio 1.3×) →
ambiguous`
- Gated by `debugAffinity` config flag

### 5. Dashboard Stats Widget
- Added to analytics overview tab when debug mode is enabled
- Metrics: total edges/nodes, resolved/ambiguous counts (%), avg
confidence, cold-start coverage, cache age, last rebuild

## Files Changed
- `cmd/server/neighbor_debug.go` — new: debug API handler, resolution
builder, cold-start coverage
- `cmd/server/neighbor_debug_test.go` — new: 7 tests for debug API
- `cmd/server/neighbor_graph.go` — added structured logging to
disambiguate(), `logFn` field, `BuildFromStoreWithLog`
- `cmd/server/neighbor_api.go` — pass debug flag through
`BuildFromStoreWithLog`
- `cmd/server/config.go` — added `DebugAffinity` config field
- `cmd/server/routes.go` — registered `/api/debug/affinity` route,
exposed `debugAffinity` in client config
- `cmd/server/types.go` — added `DebugAffinity` to
`ClientConfigResponse`
- `public/map.js` — affinity debug overlay layer with edge visualization
- `public/nodes.js` — per-node affinity debug panel
- `public/analytics.js` — dashboard stats widget
- `test-e2e-playwright.js` — 3 Playwright tests for debug UI

## Tests
- ✅ 7 Go unit tests (API shape, prefix/node filters, auth, structured
logging, cold-start coverage)
- ✅ 3 Playwright E2E tests (overlay checkbox, toggle without crash,
panel expansion)
- ✅ All existing tests pass (`go test ./cmd/server/... -count=1`)

Part of #482

---------

Co-authored-by: you <you@example.com>
 2026-04-02 23:45:03 -07:00
Kpa-clawbot e66085092e feat: neighbor affinity API endpoints (#482) — milestone 2 (#508) 
## Summary

Milestone 2 of the neighbor affinity graph (#482). Adds two API
endpoints that expose the neighbor graph built in M1 (PR #507).

### Endpoints

#### `GET /api/nodes/{pubkey}/neighbors`
Returns neighbors for a specific node with affinity scores.

**Query params:** `min_count` (default 1), `min_score` (default 0.0),
`include_ambiguous` (default true)

**Response shape:**
```json
{
  "node": "pubkey",
  "neighbors": [
    { "pubkey": "...", "prefix": "BB", "name": "...", "role": "repeater",
      "count": 847, "score": 0.95, "first_seen": "...", "last_seen": "...",
      "avg_snr": -8.2, "observers": ["obs1"], "ambiguous": false }
  ],
  "total_observations": 847
}
```

Ambiguous entries have `candidates` array; unresolved prefixes have
`unresolved: true`.

#### `GET /api/analytics/neighbor-graph`
Returns full graph summary for analytics/visualization.

**Query params:** `min_count` (default 5), `min_score` (default 0.1),
`region` (IATA code filter)

**Response shape:**
```json
{
  "nodes": [{ "pubkey": "...", "name": "...", "role": "...", "neighbor_count": 5 }],
  "edges": [{ "source": "...", "target": "...", "weight": 847, "score": 0.95, "ambiguous": false }],
  "stats": { "total_nodes": 42, "total_edges": 87, "ambiguous_edges": 3, "avg_cluster_size": 4.2 }
}
```

### Wiring
- `NeighborGraph` + `neighborMu` added to `Server` struct
- Lazy initialization: graph built on first API call, cached with 60s
TTL
- Node name/role lookups via existing `getCachedNodesAndPM()`
- Region filtering via existing `resolveRegionObservers()`

### Tests (15 tests)
- Empty graph, single neighbor, multiple neighbors (sorted by score)
- Ambiguous candidates with candidate list
- Unresolved prefix (orphan) with `unresolved: true`
- `min_count` filter, `min_score` filter, `include_ambiguous=false`
filter
- Unknown node returns 200 with empty neighbors
- Graph endpoint: empty, with edges, default min_count, ambiguous count
- Region filter (graceful when no store)
- Response shape validation (all required keys present)

All existing tests continue to pass.

Part of #482

---------

Co-authored-by: you <you@example.com>
 2026-04-02 21:30:23 -07:00