1 Commits

Include renames

Author	SHA1	Message	Date
efitenandClaude Opus 5	b614badb85	fix(packets): give each observation its own wire bytes in the detail API (#2055) ... Closes #1999. ## The defect, measured on a production instance Packet `96d716f18d885e78` on a live deployment, read from the deployed build's own API: | | | |---|---| | observations in the response | 60 | | distinct `path_json` values | 50 | | distinct `raw_hex` values returned | **1** | | distinct frames actually stored in SQLite | **51** | The contradiction the issue describes, from that same response: ``` obs 38410791 path ["58C0","1403","50C7"] -> hex 094258C01403AF37E39624E548FB7575F195A0BF ``` Three hops in the path, two path bytes in the frame. The bytes belong to the 2-hop observation and are served for all 60. Across the 3000 most recent transmissions on that database: 1977 have more than one observation and **1844 of those (93%) hold genuinely different frames**. 32659 of 36133 observations (90%) differ from their transmission's canonical bytes. This is the normal case, not an edge case. ## Cause The store deliberately does not retain `obs.RawHex`. #881 dropped it as a memory optimisation, ~98MB measured on a 1.7M-observation store, on the assumption that one content hash implies one frame. The firmware hashes payload and type independently of the relay path, so that assumption is false. Worth adding to the issue's diagnosis: all four load and ingest paths in `cmd/server/store.go` still `SELECT o.raw_hex` and scan it into `obsRawHex`, then use it nowhere — LoadAll, loadChunk, `IngestNewFromDB` and `IngestNewObservations`. The bytes are read out of SQLite and discarded, so a cold load pays the transfer for nothing. ## The fix Keeps the memory saving and reads the bytes back only where a human is looking at one packet. - **`cmd/server/db.go`** gains `ObservationRawHexForHash`: one query returning the stored frame per observation id. Two indexed lookups regardless of observation count — `transmissions.hash` through the prepared `stmtTxByHash` (`idx_transmissions_hash`), then `observations.transmission_id` (`idx_observations_transmission_id`). Guarded by `hasObsRawHex`, because #881 made the column optional and the query would be a SQL error without it. - **`cmd/server/routes.go`** backfills in `handlePacketDetail`: once per request rather than once per observation, and after the store lock is released. Bytes already present are never overwritten, and an observation with no stored frame still falls back to the transmission's. Against the acceptance list: observation bytes exposed with canonical as fallback only ✓; the store's memory optimisation untouched ✓; bounded indexed reads with no query per observation and no work under the store lock ✓; startup-loaded, newly ingested and DB-fallback details all covered, because both the store path and the DB path converge on this one backfill and both key observations by an int `id` ✓. **No frontend change is needed.** `public/packets.js` already spreads the selected observation over the packet (`{...pkt, ...currentObs}`) and already reasons about per-observation bytes: the comment there says "post-#882 per-obs raw_hex with a different path length than the top-level packet's raw_hex still gets accurate byte highlights". The client was built for this and has been receiving 60 copies of one frame. ## Tests `cmd/server/obs_raw_hex_test.go`: - the per-id mapping, with three distinct frames and a fourth observation storing none - the `hasObsRawHex` guard, so a schema without the column is not queried - the handler regression: each observation carries its own frame, the frameless one falls back to the canonical bytes, and at least three distinct frames come back across four observations — the last assertion so that a regression to repeating one frame fails, rather than passing on shape ## Verification `gofmt` clean. **Go tests were not run locally**: no cgo toolchain on this machine since #1992, and per AGENTS.md `CGO_ENABLED=0` builds a stub that proves nothing. CI is their first run. Browser validation per AGENTS.md rule 2: I verified **the defect** in a real browser and through the deployed API, with the numbers above. I could **not** validate the fix in a browser, because the change is server-side Go and is not deployed anywhere yet. Saying so rather than claiming otherwise. ## Not done - The four scan sites that fetch `o.raw_hex` and discard it are left alone. Removing the column from those query builders would stop transferring roughly ten frames per transmission on every cold load, but it touches four builders and their `scanArgs` alignment and is not needed for this defect. - `fetchResolvedPathForObs`, immediately next to this code in `enrichObsWithTx`, does run one query per observation. This change deliberately does not copy that pattern, and does not fix it either. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-22 11:08:17 +02:00