dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-09-26 04:13:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,932 Commits 83 Branches 35 Tags
a2ea18f7787c1cdbc783ec42ffabdde729e18fda
Commit Graph
15 Commits
Author SHA1 Message Date
efitenandClaude Opus 5 7f8f71e773 fix(live): reconnect when the websocket goes silent (#1074) (#2020) 
## Problem

#1074 reports that after a proxy dropped the WebSocket, live updates
only came back 8 to 10 minutes later.

The client only reconnects from `onclose` (`public/app.js:791` on
master). A half-open connection (a proxy or NAT dropping state without a
FIN reaching the browser, a laptop that slept) can keep a WebSocket OPEN
for minutes without `onclose`, and nothing retries in the meantime.

The server does ping every 30s (`cmd/server/websocket.go:252` on
master), but ping frames are answered by the browser below the page and
JS cannot observe them. The only app-level frames are packet broadcasts
(`websocket.go:337, 343, 366`), which stop on a quiet mesh. So the
client had no signal to tell a quiet mesh from a dead socket.

## Change

Server (`cmd/server/websocket.go`):
- On the existing ping tick, `writePump` also writes the text frame
`{"type":"heartbeat"}` (`:283`, bytes at `:80`). One 20-byte frame per
client per 30s, from the goroutine that already writes the ping, no hub
lock.
- The interval moves to `Hub.pingInterval` (default 30s, `:118`) so a
test can shorten it.

Client (`public/app.js`):
- Every frame refreshes `wsLastMessageAt` (`:841`). One timer
(`checkWSLiveness`, `:806`) fires at last message + `WS_STALE_MS` (75s,
one late or lost heartbeat of slack) and replaces the socket if it is
still silent. It is armed at socket creation, so a stuck handshake is
covered too.
- `dropWS` (`:796`) detaches the old socket's handlers before `close()`,
so a late close event cannot schedule a second connection.
- `connectWS` (`:818`) cancels a pending reconnect and drops the
previous socket, so the watchdog, resume checks, `onclose` and
pull-to-reconnect cannot stack sockets. Before this, `pullReconnect` on
a non-open socket left a third socket 3s later. The 3s `WS_RECONNECT_MS`
delay after `onclose` is unchanged (`:837`).
- `visibilitychange` (to visible) and `online` run the check immediately
(`:861`), because a hidden or sleeping tab's timers can run late.
- Heartbeat frames are matched by exact bytes (`:842`) and are not
pulsed or dispatched to `onWS` listeners.

Compatibility: tabs loaded before the deploy dispatch heartbeats to
their listeners until reloaded. Every current listener filters on
`msg.type`, so the visible effect is a logo pulse and a `/stats` cache
refresh every 30s.

Perf: one `Date.now()` and one string compare per WS message on the
client; one extra 20-byte write per client per 30s on the server.

## Tests

- `test-ws-stale-watchdog-1074.js`: real `app.js` in a vm with a fake
clock, timers and WebSocket. 12 tests: silence past the threshold
replaces the socket exactly once; a handshake that never opens is
replaced; heartbeats and packet traffic keep the socket; heartbeats are
not dispatched; resume and `online` after silence reconnect immediately,
with recent traffic they do not, and hiding does not trigger a check;
repeated resume events open one socket; after `onclose` only the
reconnect timer is pending; pull-to-reconnect leaves one socket. 9 of 12
fail on master. 12 of 12 source mutations (threshold, reconnect path,
detaching, timer cleanup, resume wiring, heartbeat filter) are caught.
Registered in `test-all.sh` and the deploy.yml unit step.
- `TestWritePumpSendsAppHeartbeat`: fails with a read timeout without
the heartbeat, even with pings every 20ms. `TestHubDefaultPingInterval`
pins the 30s interval that `WS_STALE_MS` assumes.
- `go test ./...` in `cmd/server` passes; gofmt and go vet are clean.

## Browser validation

On a staging instance (build `139e484e`, together with #1979's branch),
in Chrome, no console errors:

- A `{"type":"heartbeat"}` frame arrived on the open socket within the
observation window.
- Silent socket: after `ws.onmessage = null`, the page replaced the
socket after 76.2s (threshold 75s plus a 250ms poll); the old socket
ended in CLOSED, the new one OPEN.
- Normal close: `ws.close()` led to a new OPEN socket after 4.1s, and
exactly one new `WebSocket` was constructed.

## Not verified

- The reporter's proxy setup was not reproduced; that their delay was a
half-open socket is a hypothesis consistent with the symptom. Hence
`Refs`, not `Fixes`.
- Laptop sleep and the `visibilitychange` / `online` resume path were
only covered by the unit test, not in a browser.
- Behaviour under Chrome's intensive background-timer throttling and
mobile tab freezing was not measured; a frozen but healthy tab may do
one unnecessary reconnect on resume.
- Go tests were run without `-race`.

Refs #1074



## Review follow-up (commit `72e5e906`)

An independent review found no blocking bug: all data writes stay on the
write goroutine, pong-based dead-client detection still works, and no
ordering of onclose, watchdog, resume checks and pull ends with two live
sockets or none. It reproduced the silent-socket case in headless
Chromium through a blackholing TCP proxy (replacement 75.0 s after the
last frame). Changed:

1. **Startup wiring tested.** The first resume test now boots through
the page's real `DOMContentLoaded` listeners, so removing
`setupWSResumeCheck()` from startup makes it fail.
2. **Wall clock stepping back.** If the clock steps back after the last
message, the watchdog no longer re-arms for the size of the step (a 1 h
step used to delay detection by about an hour). A negative silence
reading is treated as stale, so the socket is replaced within
`WS_STALE_MS` of the step (`public/app.js:810-814`). A step in either
direction costs at most one extra reconnect on a healthy socket.
`Date.now()` stays the clock so a tab resumed after sleep is still
checked against real elapsed time.
3. **Pull-to-reconnect at once.** On an OPEN socket, pull-to-reconnect
now replaces it through `connectWS()` instead of closing it and waiting
for onclose, which took 63 s on a half-open connection in the review's
measurement (`public/app.js:927-934`). This was slow on master too; it
is safe now that `connectWS()` detaches the old socket.

Tests: 12 to 16 in `test-ws-stale-watchdog-1074.js`;
`test-pull-to-reconnect.js`, `test-pull-to-reconnect-1091.js` and
`test-live.js` pass.

Correction to the compatibility note: tabs opened before the deploy
treat the heartbeat like any other message. Besides the logo pulse,
`app.js` runs `updateNavStats` on every message and invalidates the
cached `/stats` and `/nodes` responses 5 s later; `packets.js` also
pushes every message into `pauseBuffer` unfiltered (~1310-1313), so an
old tab with Packets paused sees its counter rise by 2 per minute.
Cosmetic: heartbeats are filtered out on replay, and a reload ends it.

Not verified: real hidden-tab or mobile freeze behaviour, Firefox and
Safari, and the reporter's proxy setup.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-13 19:59:18 +02:00
efitenandClaude Opus 5 1ffaad8eb1 feat(#1794): per-IP limits and a deny list on the /ws upgrade (#1974) 
Closes #1794. Follow-up to #1793, decided **before** the upgrade because
the handshake is the resource being protected.

- Deny list of addresses and CIDRs → 403
- Per-IP concurrent connection cap → 403
- Per-IP upgrade rate limit over a rolling minute → **429**, not 403: a
temporary refusal should not read as "never come back"
- Rejection counters split by cause in `/api/stats` under `websocket`

### The decision this feature lives or dies on

Most CoreScope installs sit behind nginx, Caddy, Traefik or an ingress.
`cdn_detection.go` says so in as many words: it deliberately excludes
`X-Forwarded-For` from its CDN signals precisely because *every*
reverse-proxied install sets it. For those deployments `r.RemoteAddr` is
the proxy, `127.0.0.1` for every visitor on earth. A per-IP cap keyed on
that address protects nobody and hands the sixth legitimate browser tab
a 403. That is a self-inflicted outage wearing the costume of hardening.

So:

- **`X-Forwarded-For` is believed only from an address listed in
`webSocket.trustedProxies`.** From anywhere else it is
attacker-supplied, and trusting it would let anyone mint a fresh source
IP per connection, which is strictly worse than having no limit at all.
- **When the peer looks like a local reverse proxy and no
`trustedProxies` is set, the per-IP limits are skipped**, and one
warning names the setting that fixes it. Silently refusing real users is
the worse failure.
- **The deny list still applies there**, because it is the operator's
explicit instruction rather than an inference.

That is the answer to @mcode6726's question on the thread: it is neither
"always the socket address" nor "always the header", and the operator
decides which by naming their proxy.

### Two deliberate departures from the issue body

**`maxConnsPerIP` ships as 0 (off), not 5.** Carrier-grade NAT puts
thousands of unrelated mobile subscribers behind a single public IPv4. A
cap of 5 refuses real visitors on phones while a scraper simply rents
more addresses: all of the cost, none of the benefit.
`upgradesPerMinPerIP` ships at **30 and on**, because that one *is* safe
under CGNAT: a real client upgrades a handful of times per minute even
while reconnecting, so 30 leaves ordinary traffic untouched while
flattening a reconnect loop. A pointer type distinguishes "unset" from
an explicit `0` that turns it off.

**The default deny list is not shipped.** The thread proposed seeding 44
CIDRs for one VPS provider after a single scraper was seen at
`23.111.177.6`. I have left it out: blanket-blocking a hosting provider
by default breaks legitimate operators who host there, is undiscoverable
by the person locked out (they see a bare 403), and ages badly as ranges
get reassigned. The mechanism is here and `config.example.json` shows
exactly how to configure it, so any operator who wants that list can
have it in one line. If you want it shipped as a default anyway, that is
your call as maintainer and it is a one-line change.

### Verification

19 tests, including all five the issue specifies as TDD requirements,
each marked with the issue's own wording. Beyond those five:

- a **bare address** in the deny list works, not just CIDR form.
Operators write `1.2.3.4`, and silently ignoring that would be the worst
possible failure for a deny list: it looks configured and blocks nothing
- an unparseable deny entry is skipped and logged, not fatal. One typo
must not take the server down
- one client behind a trusted proxy does **not** exhaust another
client's budget behind the same proxy, which is the entire point of
honouring XFF
- changing a forged XFF from an untrusted peer buys no fresh budget
- `release` frees a slot and is **idempotent**, because `Unregister` can
run twice for one client and double-crediting would leak slots
- a **rejected** upgrade does not consume rate budget, or a retrying
client could never recover once its window cleared
- limits skipped for loopback and private peers; deny list applies
anyway
- a nil limiter allows everything, so a `Hub` built without
`ConfigureLimits` behaves exactly as before
- idle per-IP state is collected, while a record with a live connection
never is

Full `cmd/server` suite green, `gofmt` clean.

### Not done

- No runtime config reload; restart required. Listed as optional in the
issue.
- No `WS_DENY_IPS` env override. Also listed as optional.
- From the OWASP expansion in the first comment: `maxPayload` and the
idle/read timeout are **already in master** (`SetReadLimit`,
`SetReadDeadline`). The ping/pong heartbeat is not, and is not in this
PR either; it is a separate change to the read/write pumps and belongs
in its own review.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-06 21:11:29 +02:00
efitenSaarMesh-BotClaude Opus 5
 		40f664c587 chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941) 
Rebase of #1881 by @SaarMesh-Bot onto current master. Their three
commits are preserved, two of them cherry-picked with authorship intact;
the sweep itself had to be regenerated. Opened as a new PR rather than
force-pushing their branch.

Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2
landed as #1937.

## Why regenerated rather than merged

The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs
landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on
current master is cheaper and less error-prone than resolving 72
conflicts that are all whitespace. The drift it fixes also grew in the
meantime: 66 files now, against 72 then, but spread differently.

## The three commits

1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files.
2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet`
copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range
variable copied a `Config` embedding `sync.Once`. Cherry-picked
unchanged.
3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift
or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one
change, noted in the commit message: the ignore file pointed at
`04bc80ee`, the sweep commit on their branch, which does not exist on
this base and would make `git blame --ignore-revs-file` error. Repointed
at `d3a02599`, the sweep here.

## Verification

The claim "formatting only" is checked twice rather than asserted:

- Every changed file is byte-identical to `gofmt(previous content)`. 0
of 66 deviate.
- With line comments and all whitespace stripped, 0 of 66 files differ,
so no code outside comments changed.

14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt`
re-indents indented comment blocks to tabs and inserts a blank comment
line before them; the behavior matrix above `resolveHopWithContext` in
`cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own
output, not an edit, but it is worth naming because it makes the diff
look larger than "whitespace" suggests.

The gate was run locally exactly as the workflow runs it: `gofmt` clean,
and `go vet` clean in all 14 modules, including `cmd/ingestor` which is
what commit 2 fixes.

Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s),
`cmd/ingestor` passes except
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically
on bare master with "A required privilege is not held by the client"
(Windows symlink privilege on my host, not code).

## Sequencing

This should go last in the queue. The sweep touches 66 files, so merging
it before the remaining open Go PRs gives each of them a conflict about
nothing but formatting. After it lands the gate is active, and any PR
with drift fails CI until it runs `gofmt -w`.

Excluded from the sweep: the misnamed `Dockerfile.go`, which is a
Dockerfile that gofmt cannot parse (the workflow excludes it too), and
`docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a
spurious modification against `docs/deployment.md` and is unrelated.

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-03 18:52:03 +02:00
Kpa-clawbotandopenclaw-bot ec0ebeda2f fix(#1793): WebSocket CheckOrigin allowlist (block cross-origin scrapers) (#1795) 
## Summary

Closes the wide-open `/ws` WebSocket upgrader (`CheckOrigin: return
true`) that lets any browser origin scrape live packet data. Replaces it
with an explicit allowlist consulted from `cfg.CORSAllowedOrigins`, plus
an implicit same-origin allowance and an empty-Origin (non-browser
client) allowance.

Fixes #1793.

## Rules (`Hub.checkOrigin`)

- Empty `Origin` header → **allow** (non-browser clients; per-IP
rate/deny gating tracked separately in #1794).
- `Origin` host == request `Host` (case-insensitive) → **allow**
(same-origin).
- `Origin` matches an entry in `cfg.CORSAllowedOrigins` by exact
case-insensitive match → **allow**.
- `"*"` in `cfg.CORSAllowedOrigins` is **deliberately ignored** for
`/ws`. A startup `[ws] WARNING:` is logged once when present.
- Anything else → **reject** (gorilla returns 403).

### Deliberate divergence from CORS XHR

CORS XHR (`corsMiddleware`) still honors `"*"` for read-only
cross-origin GETs. The `/ws` upgrade does NOT, per OWASP's WebSocket
Security Cheat Sheet:

> Use an allowlist, not a denylist. Avoid wildcards or substring
matching.

—
https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html

`"*"` on the WS path would re-open the exact CSWSH/scraping vector this
PR closes, so it is rejected with a startup warning rather than silently
honored. This intentional asymmetry is documented in the updated
`_comment_corsAllowedOrigins` in `config.example.json`.

## TDD red → green

- `e5974c6a` **RED** — adds `cmd/server/websocket_checkorigin_test.go`
with five cases; `SetAllowedOrigins` introduced as an enforcement stub
so the test compiles and fails on the assertion (CI fails on this commit
by design).
- `a4791dc3` **GREEN** — implements `Hub.checkOrigin`, wires
`SetAllowedOrigins` from `main.go`, updates the config example. All
tests pass.

## Tests added (`cmd/server/websocket_checkorigin_test.go`)

- `TestCheckOriginRejectsForeignOrigin` — foreign Origin → 403
- `TestCheckOriginAllowsEmptyOrigin` — non-browser client → 101
- `TestCheckOriginAllowsSameHost` — same-origin → 101
- `TestCheckOriginAllowsAllowlistedOrigin` — exact allowlist match → 101
- `TestCheckOriginWildcardDoesNotAllowForeignOrigin` — `"*"` in
allowlist still rejects foreign origin → 403

## Files changed

- `cmd/server/websocket.go` — `Hub.allowedOrigins`, `SetAllowedOrigins`,
`checkOrigin`, wired into `Upgrader.CheckOrigin`.
- `cmd/server/main.go` — `hub.SetAllowedOrigins(cfg.CORSAllowedOrigins)`
at the single call site.
- `cmd/server/websocket_checkorigin_test.go` — new test file.
- `config.example.json` — updated `_comment_corsAllowedOrigins` to
document `/ws` gating and the `"*"` divergence.

## Out of scope (follow-up)

- **#1794** — per-IP rate limit / deny list / connection cap for
non-browser clients (which still bypass Origin because they don't send
one). Layered defense; not in this PR.

## Verification

- `go test ./cmd/server/...` — all server tests pass locally (574s).
- Preflight clean (`bash
~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`).

---------

Co-authored-by: openclaw-bot <bot@openclaw.local>
 2026-06-29 05:48:58 -07:00
efitenClaude Sonnet 4.6OpenClaw Botefiten-bot
 		caf3851ff8 feat(server): add opt-in HTTP gzip and WebSocket permessage-deflate compression (#934) 
## Summary

- Adds `"compression": {"gzip": true, "websocket": true}` config option
(both `false` by default — no behavior change)
- HTTP gzip middleware wraps the entire router; skips WebSocket upgrade
requests and clients without `Accept-Encoding: gzip`
- WebSocket permessage-deflate enabled via
`hub.upgrader.EnableCompression` when `websocket: true`
- `CompressionConfig` struct and `GZipEnabled()` /
`WSCompressionEnabled()` helpers on `Config`
- `Hub.upgrader` moved from package-level var to struct field so tests
using `NewHub()` don't need changes

## Why opt-in / off by default

Operators behind a reverse proxy that already compresses (nginx, Caddy
with `encode gzip`) should leave this off to avoid double-compression.
Only enable when the proxy does **not** compress.

## Test plan

- [x] `TestCompressionConfigDefaults` — both helpers return false when
`Compression` is nil
- [x] `TestCompressionConfigExplicitFalse` — both helpers return false
when set to false
- [x] `TestCompressionConfigEnabled` — both helpers return true when set
to true
- [x] `TestGZipMiddlewareCompresses` — response body is valid gzip,
headers set correctly
- [x] `TestGZipMiddlewareSkipsNoAcceptEncoding` — passthrough when
client doesn't send Accept-Encoding: gzip
- [x] `TestGZipMiddlewareSkipsWebSocket` — WebSocket upgrades are never
gzip-wrapped

All 6 tests pass (`go test ./...` in `cmd/server`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: OpenClaw Bot <bot@openclaw.local>
Co-authored-by: efiten-bot <bot@efiten.dev>
 2026-05-21 11:39:49 -07:00
Kpa-clawbotyouKpa-clawbot
 		f87eb3601c fix: graceful container shutdown for reliable deployments (#453) 
## Summary

Fixes #450 — staging deployment flaky due to container not shutting down
cleanly.

## Root Causes

1. **Server never closed DB on shutdown** — SQLite WAL lock held
indefinitely, blocking new container startup
2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills
connections instead of draining them
3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM
then immediately SIGKILL (default 10s is often not enough for WAL
checkpoint)
4. **Supervisor didn't forward SIGTERM** — missing
`stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of
graceful shutdown
5. **Deploy scripts used default `docker stop` timeout** — only 10s
grace period

## Changes

### Go Server (`cmd/server/`)
- **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s
context timeout — drains in-flight requests before closing
- **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway`
frames to all connected clients
- **DB close on shutdown**: Explicitly closes DB after HTTP server stops
(was never closed before)
- **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close
— flushes WAL to main DB file and removes WAL/SHM lock files

### Go Ingestor (`cmd/ingestor/`)
- **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method,
called before `Close()`
- **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight
messages to drain

### Docker Compose (all 4 variants)
- Added `stop_grace_period: 30s` and `stop_signal: SIGTERM`

### Supervisor Configs (both variants)
- Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor
programs

### Deploy Scripts
- `deploy-staging.sh`: `docker stop -t 30` with explicit grace period
- `deploy-live.sh`: `docker stop -t 30` with explicit grace period

## Shutdown Sequence (after fix)

1. Docker sends SIGTERM to supervisord (PID 1)
2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s
each)
3. Server: stops poller → drains HTTP (15s) → closes WS clients →
checkpoints WAL → closes DB
4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL →
closes DB
5. Docker waits up to 30s total before SIGKILL

## Tests

All existing tests pass:
- `cd cmd/server && go test ./...` ✅
- `cd cmd/ingestor && go test ./...` ✅

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 12:19:20 -07:00
Kpa-clawbotandCopilot 738d5fef39 fix: poller uses store max IDs to prevent replaying entire DB 
When GetMaxTransmissionID() fails silently (e.g., corrupted DB returns 0
from COALESCE), the poller starts from ID 0 and replays the entire
database over WebSocket — broadcasting thousands of old packets per second.

Fix: after querying the DB, use the in-memory store's MaxTransmissionID
and MaxObservationID as a floor. Since Load() already read the full DB
successfully, the store has the correct max IDs.

Root cause discovered on staging: DB corruption caused MAX(id) query to
fail, returning 0. Poller log showed 'starting from transmission ID 0'
followed by 1000-2000 broadcasts per tick walking through 76K rows.

Also adds MaxObservationID() to PacketStore for observation cursor safety.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-31 23:28:56 -07:00
Kpa-clawbotandCopilot ab03b142f5 fix: per-observation WS broadcast for live view starburst — fixes #237 
IngestNewFromDB now broadcasts one message per observation (not per
transmission). IngestNewObservations also broadcasts late arrivals.
Tests verify multi-observer packets produce multiple WS messages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-29 08:32:37 -07:00
Kpa-clawbotandCopilot df63efa78d fix: poll new observations for existing transmissions (fixes #174) 
The poller only queried WHERE t.id > sinceID, which missed new
observations added to transmissions already in the store. The trace
page was correct because it always queries the DB directly.

Add IngestNewObservations() that polls observations by o.id watermark,
adds them to existing StoreTx entries, re-picks best observation, and
invalidates analytics caches. The Poller now tracks both lastTxID and
lastObsID watermarks.

Includes tests for v3, v2, dedup, best-path re-pick, and
GetMaxObservationID.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 17:26:26 -07:00
Kpa-clawbotandCopilot f55a3454aa feat(go): replace map[string]interface{} with typed Go structs in route handlers 
Phase 1: Create cmd/server/types.go with ~80 typed response structs
matching all proto definitions. Every API response shape is now a
compile-time checked struct.

Phase 2: Rewire all route handlers in routes.go to construct typed
structs instead of map[string]interface{} for response building:
- /api/stats -> StatsResponse
- /api/health -> HealthResponse
- /api/perf -> PerfResponse
- /api/config/* -> typed config responses
- /api/nodes/* -> NodeListResponse, NodeDetailResponse, etc.
- /api/packets/* -> PacketListResponse, PacketDetailResponse
- /api/analytics/* -> RFAnalyticsResponse, TopologyResponse, etc.
- /api/observers/* -> ObserverListResponse, ObserverResp
- /api/channels/* -> ChannelListResponse, ChannelMessagesResponse
- /api/traces/* -> TraceResponse
- /api/resolve-hops -> ResolveHopsResponse
- /api/iata-coords -> IataCoordsResponse (typed IataCoord)
- /api/audio-lab/buckets -> AudioLabBucketsResponse
- WebSocket broadcast -> WSMessage struct
- SlowQuery tracking -> SlowQuery struct (was map)

Phase 3 (partial): Add typed store/db methods:
- PacketStore.GetCacheStatsTyped() -> CacheStats
- PacketStore.GetPerfStoreStatsTyped() -> PerfPacketStoreStats
- DB.GetDBSizeStatsTyped() -> SqliteStats

Remaining map usage is in store/db data flow (PacketResult.Packets
still uses maps) — these will be addressed in a follow-up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 15:17:21 -07:00
Kpa-clawbotandCopilot fc494962d1 fix(go): add broadcast diagnostic logging, rebuild fixes stale deployment 
The Go staging packets page wasn't live-updating because the deployed
binary was stale (built before the #162 fix). Rebuilding from current
source fixed the issue — broadcasts now fire correctly.

Added two permanent diagnostic log lines:
- [poller] IngestNewFromDB: logs when new transmissions are found
- [broadcast] sending N packets to M clients: logs each broadcast batch

These log lines make it easy to verify the broadcast pipeline is working
and would have caught this stale-deployment issue immediately.

Verified on VM: WS clients receive packets with nested 'packet' field,
all Go tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 14:15:19 -07:00
Kpa-clawbotandCopilot 95afaf2f0d fix(go): add nested packet field to WS broadcast, fixes #162 
The frontend packets.js filters WS messages with m.data?.packet and
extracts m.data.packet for live rendering. Node's server.js includes
a packet sub-object (packet: fullPacket) in the broadcast data, but
Go's IngestNewFromDB built the data flat without a nested packet field.

This caused the Go staging packets page to never live-update via WS
even though messages were being sent — they were silently filtered out
by packets.js.

Fix: build the packet fields map separately, then create the broadcast
map with both top-level fields (for live.js) and nested packet (for
packets.js). Also fixes the fallback DB-direct poller path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 13:31:37 -07:00
Kpa-clawbotandCopilot afe16db960 feat(go-server): in-memory packet store — port of packet-store.js 
Streams transmissions + observations from SQLite at startup into
5 indexed in-memory structures. QueryPackets and QueryGroupedPackets
now serve from RAM (<10ms) instead of hitting SQLite (2.3s).

- store.go: PacketStore with byHash, byTxID, byObsID, byObserver, byNode indexes
- main.go: create + load store at startup
- routes.go: dispatch to store for packet/stats endpoints
- websocket.go: poller ingests new transmissions into store

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 08:52:07 -07:00
Kpa-clawbotandCopilot b2e6c8105b fix: handle WebSocket upgrade at root path (client connects to ws://host/) 
Node.js upgrades WS at /, Go was only at /ws. Now the static file
handler checks for Upgrade header first and routes to WebSocket.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 01:25:35 -07:00
Kpa-clawbotandCopilot 742ed86596 feat: add Go web server (cmd/server/) — full API + WebSocket + static files 
35+ REST endpoints matching Node.js server, WebSocket broadcast,
static file serving with SPA fallback, config.json support.
Uses modernc.org/sqlite (pure Go, no CGO required).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 01:16:59 -07:00