Files
meshcore-analyzer/cmd/server
ZachandClaude Opus 5.5 2ecc3d1e72 fix(server): never lose rows committed during startup or a failed poll (#2408)
## Summary

Two pre-existing ways a running server **permanently** missed data. The
database was correct; the server's in-memory store was not, and stayed
wrong until the next restart.

Found while comparing several server replicas side by side in #2396: two
replicas that restarted mid-ingest each lacked one transmission and its
12 observations. Split out of #2396 at review's request, because it
affects every SQLite install.

## The bugs

1. **Startup window.** The poller took its starting cursors from a fresh
`MAX(id)` *after* the startup load finished. A row committed between the
load's last chunk snapshot and that query ended up in neither the store
nor any later poll. Its later observations were then dropped as orphans.
2. **Failed poll.** A failed observation read still advanced the cursor
to its upper bound, skipping up to 500 observations for good.

## The fix

- `NewPacketStore` records the transmission and observation high-water
marks **before** the load, as the advert-evidence and rewrite feeds
already did.
  - The loaded max transmission id stays as a floor.
- The loaded max observation id no longer is one: a chunk-1 transmission
can gain an observation whose id is *below* one that chunk 2 loaded.
  - Rows read twice are deduplicated and broadcast at most once.
- A failed read keeps the cursor and retries on the next tick, logging
once per failure streak.
- Comments and a test pin why an observation of a transmission whose
chunk has not merged yet cannot be skipped: the poller's evidence
refresh waits on that chunk's lock.

## Performance

- **Startup:** two extra indexed `MAX(id)` queries, which are
primary-key lookups.
- **Poll loop:** unchanged complexity. The only new per-tick work is a
dedupe check on rows read twice, a map lookup by id (O(1)). Re-reads
happen only for rows in the startup window, which is bounded by what was
committed during the load, and on a retry after a failed read.

## Testing

`cmd/server/startup_poll_gap_test.go` seeds the ingestor's own DDL plus
`dbschema.Apply`:

| Test | What it pins |
|------|--------------|
| `TestStartupLoadPollerGap_SQLite` | A transmission and observations
committed between the load's snapshot and the poller's start reach the
store |
| `TestStartupLoadPollerGapMultiChunk_SQLite` | Same, across 3 chunks,
including an observation id below the max loaded one |
| `TestStartupObsDuringChunkMerge_SQLite` | An observation of a
transmission whose chunk has not merged yet is not skipped |
| `TestPollerRetriesFailedRead_SQLite` | A failed observation read does
not advance the cursor |

**Red/green:** against master's `store.go`, `websocket.go`,
`chunked_load.go` and `db.go`, the first, second and fourth tests fail
with the rows missing, e.g.:

```
tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever)
obs 3, committed while the observation read failed, never reached the store: the cursor passed it
```

The third depends on test hooks this change adds, so it cannot run
against the old code. `go vet` and `go test` pass on all modules, with
gofmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`.

Recommended **priority** (not a dependency):

1. **#2410** — musl 8 MiB thread stacks. A public, unauthenticated GET
can SIGSEGV the whole server on the shipped image, so merge this first.
2. **#2408** — never lose rows during startup / a failed poll.
3. **#2407** — explicit `observers.idx` primary key.
4. **#2409** — content-hash migration moved into the ingestor.

After these land, #2396 (Postgres) will be rebased onto master.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:31:45 +01:00
..