Closes#1999.
## The defect, measured on a production instance
Packet `96d716f18d885e78` on a live deployment, read from the deployed
build's own API:
| | |
|---|---|
| observations in the response | 60 |
| distinct `path_json` values | 50 |
| distinct `raw_hex` values returned | **1** |
| distinct frames actually stored in SQLite | **51** |
The contradiction the issue describes, from that same response:
```
obs 38410791 path ["58C0","1403","50C7"] -> hex 094258C01403AF37E39624E548FB7575F195A0BF
```
Three hops in the path, two path bytes in the frame. The bytes belong to
the 2-hop observation and are served for all 60.
Across the 3000 most recent transmissions on that database: 1977 have
more than one observation and **1844 of those (93%) hold genuinely
different frames**. 32659 of 36133 observations (90%) differ from their
transmission's canonical bytes. This is the normal case, not an edge
case.
## Cause
The store deliberately does not retain `obs.RawHex`. #881 dropped it as
a memory optimisation, ~98MB measured on a 1.7M-observation store, on
the assumption that one content hash implies one frame. The firmware
hashes payload and type independently of the relay path, so that
assumption is false.
Worth adding to the issue's diagnosis: all four load and ingest paths in
`cmd/server/store.go` still `SELECT o.raw_hex` and scan it into
`obsRawHex`, then use it nowhere — LoadAll, loadChunk, `IngestNewFromDB`
and `IngestNewObservations`. The bytes are read out of SQLite and
discarded, so a cold load pays the transfer for nothing.
## The fix
Keeps the memory saving and reads the bytes back only where a human is
looking at one packet.
- **`cmd/server/db.go`** gains `ObservationRawHexForHash`: one query
returning the stored frame per observation id. Two indexed lookups
regardless of observation count — `transmissions.hash` through the
prepared `stmtTxByHash` (`idx_transmissions_hash`), then
`observations.transmission_id` (`idx_observations_transmission_id`).
Guarded by `hasObsRawHex`, because #881 made the column optional and the
query would be a SQL error without it.
- **`cmd/server/routes.go`** backfills in `handlePacketDetail`: once per
request rather than once per observation, and after the store lock is
released. Bytes already present are never overwritten, and an
observation with no stored frame still falls back to the transmission's.
Against the acceptance list: observation bytes exposed with canonical as
fallback only ✓; the store's memory optimisation untouched ✓; bounded
indexed reads with no query per observation and no work under the store
lock ✓; startup-loaded, newly ingested and DB-fallback details all
covered, because both the store path and the DB path converge on this
one backfill and both key observations by an int `id` ✓.
**No frontend change is needed.** `public/packets.js` already spreads
the selected observation over the packet (`{...pkt, ...currentObs}`) and
already reasons about per-observation bytes: the comment there says
"post-#882 per-obs raw_hex with a different path length than the
top-level packet's raw_hex still gets accurate byte highlights". The
client was built for this and has been receiving 60 copies of one frame.
## Tests
`cmd/server/obs_raw_hex_test.go`:
- the per-id mapping, with three distinct frames and a fourth
observation storing none
- the `hasObsRawHex` guard, so a schema without the column is not
queried
- the handler regression: each observation carries its own frame, the
frameless one falls back to the canonical bytes, and at least three
distinct frames come back across four observations — the last assertion
so that a regression to repeating one frame fails, rather than passing
on shape
## Verification
`gofmt` clean. **Go tests were not run locally**: no cgo toolchain on
this machine since #1992, and per AGENTS.md `CGO_ENABLED=0` builds a
stub that proves nothing. CI is their first run.
Browser validation per AGENTS.md rule 2: I verified **the defect** in a
real browser and through the deployed API, with the numbers above. I
could **not** validate the fix in a browser, because the change is
server-side Go and is not deployed anywhere yet. Saying so rather than
claiming otherwise.
## Not done
- The four scan sites that fetch `o.raw_hex` and discard it are left
alone. Removing the column from those query builders would stop
transferring roughly ten frames per transmission on every cold load, but
it touches four builders and their `scanArgs` alignment and is not
needed for this defect.
- `fetchResolvedPathForObs`, immediately next to this code in
`enrichObsWithTx`, does run one query per observation. This change
deliberately does not copy that pattern, and does not fix it either.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
