Addresses the four P0+P1 firmware reconciliation gaps from the umbrella
audit (issue #1279). RED commit: `0a4c084e` (asserts on stub returns;
all 13 assertions fail). GREEN commit: `13867681`.
## What's in this PR
### P0 — silently dropped data
- **#1 GRP_DATA (0x06) decoder.** Outer envelope is the same shape as
GRP_TXT (`channel_hash(1)+MAC(2)+ciphertext`) per
`firmware/src/helpers/BaseChatMesh.cpp:476,500`. Factored
`decryptChannelBlock(...)` helper used by both 5 and 6. When a channel
key matches, the inner is parsed per
`firmware/src/helpers/BaseChatMesh.cpp:382-385` as `data_type(uint16 LE)
+ data_len(1) + blob(data_len)`. Surfaces `{channelHash, MAC, dataType,
dataLen, decryptedBlob}` on decrypt or `{channelHash, MAC,
encryptedData}` otherwise. Server-side decoder surfaces envelope only
(no key store).
- **#2 MULTIPART (0x0A) decoder.** Per `firmware/src/Mesh.cpp:289`,
byte0 = `(remaining<<4) | inner_type`. When `inner_type ==
PAYLOAD_TYPE_ACK (0x03)`, next 4 bytes are the LE ack_crc per
`firmware/src/Mesh.cpp:292-307`. Surfaces `{remaining, innerType,
innerTypeName, innerAckCrc | innerPayload}`.
### P1 — mis-classified / opaque
- **#3 `advertRole()` raw-type fix.** Per
`firmware/src/helpers/AdvertDataHelpers.h:7-12`, ADV_TYPE_NONE = 0 and
5-15 are FUTURE. The previous boolean fallback collapsed both into
`"companion"`, silently relabelling unknown/reserved types. New
behaviour: type 0 → `none`, 1 → `companion`, 2-4 →
`repeater`/`room`/`sensor`, 5-15 → `type-N`. `ValidateAdvert` accepts
the new labels.
- **#4 CONTROL (0x0B) byte0 flags + length.** Per
`firmware/src/Mesh.cpp:69` + `createControlData` at `Mesh.cpp:609`,
byte0 high-bit marks the zero-hop direct subset. Surfaces `{ctrlFlags,
ctrlZeroHop, ctrlLength}`.
### Drift fix
- `cmd/server/store.go` `payloadTypeNames` now includes `6: GRP_DATA`
and `10: MULTIPART` (previously omitted; canonical decoder map already
had them).
## Lockstep & TDD
Both `cmd/ingestor/decoder.go` and `cmd/server/decoder.go` updated in
the same commits — same wire-vector tests live in both packages
(`cmd/{ingestor,server}/issue1279_test.go`). Per-item RED→GREEN visible
in `git log`.
| Item | Tests | RED proof |
|---|---|---|
| #1 GRP_DATA | ingestor: NoKey + DecryptedInner; server: Envelope | 6
assertions failed pre-impl |
| #2 MULTIPART | ingestor + server: Ack + NonAck | 8 assertions failed
pre-impl |
| #3 advertRole | ingestor + server: 7-row table | 3 assertions failed
pre-impl |
| #4 CONTROL | ingestor + server: ZeroHop + MultiHop | 6 assertions
failed pre-impl |
## What's NOT in this PR
The umbrella issue lists P2 items that ship in follow-up PRs:
- Live + compare legend entries for the long tail of newly-named types
(#1274 + others).
- TransportCodes UI surface + filter grammar.
- feat1/feat2 capability badges.
- `payloadTypeNames` consolidation across server/ingestor
(drift-prevention).
Leave the umbrella open after this merges.
Refs #1279
---------
Co-authored-by: OpenClaw Bot <bot@openclaw.local>
**RED commit:** `65d9f57b` (CI run will appear at
https://github.com/Kpa-clawbot/CoreScope/actions after PR opens)
Fixes#1211
## Root cause
`decodePath()` returns `bytesConsumed = hash_size * hash_count` where
both come straight from the wire-supplied `pathByte` (upper 2 bits →
`hash_size`, lower 6 bits → `hash_count`). Max claimable: 4 × 63 = 252
bytes.
A malformed packet on the wire claimed `pathByte=0xF6` (hash_size=4,
hash_count=54 → 216 path bytes) inside a 15-byte buffer. The inner
hop-extraction loop in `decodePath` did break early on overflow — but
`bytesConsumed` was still returned at face value (216). `DecodePacket`
then did `offset += 216` (offset=218) and `payloadBuf := buf[offset:]`
panicked with the prod-observed signature:
```
runtime error: slice bounds out of range [218:15]
```
The handler-level `defer/recover` at `cmd/ingestor/main.go:258-263`
caught it, but the message was silently dropped with no usable
diagnostic.
## Fix
Add a `if offset > len(buf)` guard at BOTH decoder sites (same pattern,
same panic potential):
- `cmd/ingestor/decoder.go` — DecodePacket after decodePath
- `cmd/server/decoder.go` — DecodePacket after decodePath
Return a descriptive error citing the claimed length and pathByte hex so
operators can reproduce.
Also: `cmd/ingestor/main.go` decode-error log now includes `topic`,
`observer`, and `rawHexLen` so future malformed packets are reproducible
without needing to attach a debugger.
## Tests (TDD red → green)
Both packages got two new tests:
- **`TestDecodePacketBoundsFromWire_Issue1211`** — feeds the exact wire
shape from the prod log (`pathByte=0xF6` inside a 15-byte buf). Asserts
`DecodePacket` does NOT panic and returns an error.
- **`TestDecodePacketFuzzTruncated_Issue1211`** — sweeps every `(header,
pathByte)` combination with tails 0..19 bytes (≈1.3M inputs). Asserts
zero panics.
### Red commit proof
On commit `65d9f57b` (RED), both tests fail with the panic:
```
=== RUN TestDecodePacketBoundsFromWire_Issue1211
decoder_test.go:1996: DecodePacket panicked on malformed input: runtime error: slice bounds out of range [218:15]
--- FAIL: TestDecodePacketBoundsFromWire_Issue1211 (0.00s)
=== RUN TestDecodePacketFuzzTruncated_Issue1211
decoder_test.go:2010: DecodePacket panicked during fuzz: runtime error: slice bounds out of range [3:2]
--- FAIL: TestDecodePacketFuzzTruncated_Issue1211 (0.01s)
```
On commit `7a6ae52c` (GREEN), full suites pass:
- `cmd/ingestor`: `ok 53.988s`
- `cmd/server`: `ok 29.456s`
## Acceptance criteria
- [x] Identify the slice op producing `[218:15]` — `payloadBuf :=
buf[offset:]` in `DecodePacket` (decoder.go), where `offset` had been
advanced by an unchecked `bytesConsumed` from `decodePath()`.
- [x] Bounds check added at the identified site(s) — both ingestor and
server decoders.
- [x] Test with crafted payload (length-field > remaining buffer) —
`TestDecodePacketBoundsFromWire_Issue1211`.
- [x] Log topic, observer ID, payload byte length on drop — updated
`MQTT [%s] decode error` log line.
- [x] Existing tests stay green — confirmed both packages.
## Out of scope
Reconnect-after-disconnect (#1212) — handled by a separate subagent.
This PR touches NO reconnect logic.
---------
Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: corescope-bot <bot@corescope>
## Summary
Parse and display per-hop SNR values from TRACE packets in the Packet
Byte Breakdown panel.
## Changes
### Backend (`cmd/server/decoder.go`)
- Added `SNRValues []float64` field to Payload struct
(`json:"snrValues,omitempty"`)
- In the TRACE-specific block, extract SNR from header path bytes before
they're overwritten with route hops
- Each header path byte is `int8(SNR_dB * 4.0)` per firmware — decode by
dividing by 4.0
### Frontend (`public/packets.js`)
- Added "SNR Path" section in `buildFieldTable()` showing per-hop SNR
values in dB when packet type is TRACE
- Added TRACE-specific payload rendering (trace tag, auth code, flags
with hash_size, route hops)
## TDD
- Red commit: `4dba4e8` — test asserts `Payload.SNRValues` field
(compile fails, field doesn't exist)
- Green commit: `5a496bd` — implementation passes all tests
## Testing
- `go test ./...` passes (all existing + 2 new TRACE SNR tests)
- No frontend test changes needed (no existing TRACE UI tests; rendering
is additive)
Fixes#979
---------
Co-authored-by: you <you@example.com>
## Symptom
The colored byte strip in the packet detail pane is offset from the
labeled byte breakdown below it. Off by N bytes where N is the
difference between the top-level packet's path length and the displayed
observation's path length.
## Root cause
Server computes `breakdown.ranges` once from the top-level packet's
raw_hex (in `BuildBreakdown`) and ships it in the API response. After
#882 we render each observation's own raw_hex, but we keep using the
top-level breakdown — so a 7-hop top-level packet shipped "Path: bytes
2-8", and when we rendered an 8-hop observation we coloured 7 of the 8
path bytes and bled into the payload.
The labeled rows below (which use `buildFieldTable`) parse the displayed
raw_hex on the client, so they were correct — they just didn't match the
strip above.
## Fix
Port `BuildBreakdown()` to JS as `computeBreakdownRanges()` in `app.js`.
Use it in `renderDetail()` from the actually-rendered (per-obs) raw_hex.
## Test
Manually verified the JS function output matches the Go implementation
for FLOOD/non-transport, transport, ADVERT, and direct-advert (zero
hops) cases.
Closes nothing (caught in post-tag bug bash).
---------
Co-authored-by: you <you@example.com>
## Problem
Per-observation `path_json` disagrees with `raw_hex` path section for
TRACE packets.
**Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡`
- `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE
payload)
- `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header)
## Root Cause
`DecodePacket` correctly parses TRACE packets by replacing `path.Hops`
with hop IDs from the payload's `pathData` field (the actual route).
However, the header path bytes for TRACE packets contain **SNR values**
(one per completed hop), not hop IDs.
`BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which
for TRACE packets contained the payload-derived hops — not the header
path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex`
to describe completely different paths.
## Fix
- Added `DecodePathFromRawHex(rawHex)` — extracts header path hops
directly from raw hex bytes, independent of any TRACE payload
overwriting.
- `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of
using `decoded.Path.Hops`, guaranteeing `path_json` always matches the
`raw_hex` path section.
## Tests (8 new)
**`DecodePathFromRawHex` unit tests:**
- hash_size 1, 2, 3, 4
- zero-hop direct packets
- transport route (4-byte transport codes before path)
**`BuildPacketData` integration tests:**
- TRACE packet: asserts path_json matches raw_hex header path (not
payload hops)
- Non-TRACE packet: asserts path_json matches raw_hex header path
All existing tests continue to pass (`go test ./...` for both ingestor
and server).
Fixes#886
---------
Co-authored-by: you <you@example.com>
## Problem
The firmware computes packet content hash as:
```
SHA256(payload_type_byte + [path_len for TRACE] + payload)
```
Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type
bits (2-5).
CoreScope was using the **full header byte** in its hash computation,
which includes route type bits (0-1) and version bits (6-7). This meant
the same logical packet produced different content hashes depending on
route type — breaking dedup and packet lookup.
**Firmware reference:** `Packet.cpp::calculatePacketHash()` uses
`getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) &
PH_TYPE_MASK`.
## Fix
- Extract only payload type bits: `payloadType := (headerByte >> 2) &
0x0F`
- Include `path_len` byte in hash for TRACE packets (matching firmware
behavior)
- Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go`
## Tests Added
- **Route type independence:** Same payload with FLOOD vs DIRECT route
types produces identical hash
- **TRACE path_len inclusion:** TRACE packets with different `path_len`
produce different hashes
- **Firmware compatibility:** Hash output matches manual computation of
firmware algorithm
## Migration Impact
Existing packets in the DB have content hashes computed with the old
(incorrect) formula. Options:
1. **Recompute hashes** via migration (recommended for clean state)
2. **Dual lookup** — check both old and new hash on queries (backward
compat)
3. **Accept the break** — old hashes become stale, new packets get
correct hashes
Recommend option 1 (migration) as a follow-up. The volume of affected
packets depends on how many distinct route types were seen for the same
logical packet.
Fixes#786
---------
Co-authored-by: you <you@example.com>
## Summary
TRACE packets encode their route hash size in the flags byte (`flags &
0x03`), not the header path byte. The decoder was using `path.HashSize`
from the header, which could be wrong or zero for direct-route TRACEs,
producing incorrect hop counts in `path_json`.
## Protocol Note
Per firmware, TRACE packets are **always direct-routed** (route_type 2 =
DIRECT, or 3 = TRANSPORT_DIRECT). FLOOD-routed TRACEs (route_type 1) are
anomalous — firmware explicitly rejects TRACE via flood. The decoder
handles these gracefully without crashing.
## Changes
**`cmd/server/decoder.go` and `cmd/ingestor/decoder.go`:**
- Read `pathSz` from TRACE flags byte: `(traceFlags & 0x03) + 1`
(0→1byte, 1→2byte, 2→3byte)
- Use `pathSz` instead of `path.HashSize` for splitting TRACE payload
path data into hops
- Update `path.HashSize` to reflect the actual TRACE path size
- Added `HopsCompleted` field to ingestor `Path` struct for parity with
server
- Updated comments to clarify TRACE is always direct-routed per firmware
**`cmd/server/decoder_test.go` — 5 new tests:**
- `TraceFlags1_TwoBytePathSz`: flags=1 → 2-byte hashes via DIRECT route
- `TraceFlags2_ThreeBytePathSz`: flags=2 → 3-byte hashes via DIRECT
route
- `TracePathSzUnevenPayload`: payload not evenly divisible by path_sz
- `TraceTransportDirect`: route_type=3 with transport codes + TRACE path
parsing
- `TraceFloodRouteGraceful`: anomalous FLOOD+TRACE handled without crash
All existing TRACE tests (flags=0, 1-byte hashes) continue to pass.
Fixes#731
---------
Co-authored-by: you <you@example.com>
For BYOP mode in the packet analyzer, perform signature validation on
advert packets and display whether successful or not. This is added as
we observed many corrupted advert packets that would be easily
detectable as such if signature validation checks were performed.
At present this MR is just to add this status in BYOP mode so there is
minimal impact to the application and no performance penalty for having
to perform these checks on all packets. Moving forward it probably makes
sense to do these checks on all advert packets so that corrupt packets
can be ignored in several contexts (like node lists for example).
Let me know what you think and I can adjust as needed.
---------
Co-authored-by: you <you@example.com>
## Summary
TRACE packets on the live map previously animated the **full intended
route** regardless of how far the trace actually reached. This made it
impossible to distinguish a completed route from a failed one —
undermining the primary diagnostic purpose of trace packets.
## Changes
### Backend — `cmd/server/decoder.go`
- Added `HopsCompleted *int` field to the `Path` struct
- For TRACE packets, the header path contains SNR bytes (one per hop
that actually forwarded). Before overwriting `path.Hops` with the full
intended route from the payload, we now capture the header path's
`HashCount` as `hopsCompleted`
- This field is included in API responses and WebSocket broadcasts via
the existing JSON serialization
### Frontend — `public/live.js`
- For TRACE packets with `hopsCompleted < totalHops`:
- Animate only the **completed** portion (solid line + pulse)
- Draw the **unreached** remainder as a dashed/ghosted line (25%
opacity, `6,8` dash pattern) with ghost markers
- Dashed lines and ghost markers auto-remove after 10 seconds
- When `hopsCompleted` is absent or equals total hops, behavior is
unchanged
### Tests — `cmd/server/decoder_test.go`
- `TestDecodePacket_TraceHopsCompleted` — partial completion (2 of 4
hops)
- `TestDecodePacket_TraceNoSNR` — zero completion (trace not forwarded
yet)
- `TestDecodePacket_TraceFullyCompleted` — all hops completed
## How it works
The MeshCore firmware appends an SNR byte to `pkt->path[]` at each hop
that forwards a TRACE packet. The count of these SNR bytes (`path_len`)
indicates how far the trace actually got. CoreScope's decoder already
parsed the header path, but the TRACE-specific code overwrote it with
the payload hops (full intended route) without preserving the progress
information. Now we save that count first.
Fixes#651
---------
Co-authored-by: you <you@example.com>
## Fix: Zero-hop DIRECT packets report bogus hash_size
Closes#649
### Problem
When a DIRECT packet has zero hops (pathByte lower 6 bits = 0), the
generic `hash_size = (pathByte >> 6) + 1` formula produces a bogus value
(1-4) instead of 0/unknown. This causes incorrect hash size displays and
analytics for zero-hop direct adverts.
### Solution
**Frontend (JS):**
- `packets.js` and `nodes.js` now check `(pathByte & 0x3F) === 0` to
detect zero-hop packets and suppress bogus hash_size display.
**Backend (Go):**
- Both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` reset
`HashSize=0` for DIRECT packets where `pathByte & 0x3F == 0` (hash_count
is zero).
- TRACE packets are excluded since they use hashSize to parse hop data
from the payload.
- The condition uses `pathByte & 0x3F == 0` (not `pathByte == 0x00`) to
correctly handle the case where hash_size bits are non-zero but
hash_count is zero — matching the JS frontend approach.
### Testing
**Backend:**
- Added 4 tests each in `cmd/server/decoder_test.go` and
`cmd/ingestor/decoder_test.go`:
- DIRECT + pathByte 0x00 → HashSize=0 ✅
- DIRECT + pathByte 0x40 (hash_size bits set, hash_count=0) → HashSize=0
✅
- Non-DIRECT + pathByte 0x00 → HashSize=1 (unchanged) ✅
- DIRECT + pathByte 0x01 (1 hop) → HashSize=1 (unchanged) ✅
- All existing tests pass (`go test ./...` in both cmd/server and
cmd/ingestor)
**Frontend:**
- Verified hash size display is suppressed for zero-hop direct adverts
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
## Summary
- `BuildBreakdown` was never ported from the deleted Node.js
`decoder.js` to Go — the server has returned `breakdown: {}` since the
Go migration (commit `742ed865`), so `createColoredHexDump()` and
`buildHexLegend()` in the frontend always received an empty `ranges`
array and rendered everything as monochrome
- Implemented `BuildBreakdown()` in `decoder.go` — computes labeled byte
ranges matching the frontend's `LABEL_CLASS` map: `Header`, `Transport
Codes`, `Path Length`, `Path`, `Payload`; ADVERT packets get sub-ranges:
`PubKey`, `Timestamp`, `Signature`, `Flags`, `Latitude`, `Longitude`,
`Name`
- Wired into `handlePacketDetail` (was `struct{}{}`)
- Also adds per-section color classes to the field breakdown table
(`section-header`, `section-transport`, `section-path`,
`section-payload`) so the table rows get matching background tints
## Test plan
- [x] Open any packet detail pane — hex dump should show color-coded
sections (red header, orange path length, blue transport codes, green
path hops, yellow/colored payload)
- [x] Legend below action buttons should appear with color swatches
- [x] ADVERT packets: PubKey/Timestamp/Signature/Flags each get their
own distinct color
- [x] Field breakdown table section header rows should be tinted per
section
- [x] 8 new Go tests: all pass
Closes#329🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#276
## Root cause
TRACE packets store hop IDs in the payload (bytes 9+) rather than in the
header path field. The header path field is overloaded in TRACE packets
to carry RSSI values instead of repeater IDs (as noted in the issue
comments). This meant `Path.Hops` was always empty for TRACE packets —
the raw bytes ended up as an opaque `PathData` hex string with no
structure.
The hashSize encoded in the header path byte (bits 6–7) is still valid
for TRACE and is used to split the payload path bytes into individual
hop prefixes.
## Fix
After decoding a TRACE payload, if `PathData` is non-empty, parse it
into individual hops using `path.HashSize`:
```go
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
path.Hops = append(path.Hops, ...)
}
}
}
```
Applied to both `cmd/ingestor/decoder.go` and `cmd/server/decoder.go`.
## Verification
Packet from the issue: `260001807dca00000000007d547d`
| | Before | After |
|---|---|---|
| `Path.Hops` | `[]` | `["7D", "54", "7D"]` |
| `Path.HashCount` | `0` | `3` |
New test `TestDecodeTracePathParsing` covers this exact packet.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Match the C++ firmware wire format (Packet::writeTo/readFrom):
1. Field order: transport codes are parsed BEFORE path_length byte,
matching firmware's header → transport_codes → path_len → path → payload
2. ACK payload: just 4-byte CRC checksum, not dest+src+ackHash.
Firmware createAck() writes only ack_crc (4 bytes).
3. TRACE payload: tag(4) + authCode(4) + flags(1) + pathData,
matching firmware createTrace() and onRecvPacket() TRACE handler.
4. ADVERT features: parse feat1 (0x20) and feat2 (0x40) optional
2-byte fields between location and name, matching AdvertDataBuilder
and AdvertDataParser in the firmware.
5. Transport code naming: code1/code2 instead of nextHop/lastHop,
matching firmware's transport_codes[0]/transport_codes[1] naming.
Fixes applied to both cmd/ingestor/decoder.go and cmd/server/decoder.go.
Tests updated to match new behavior.
#184: Strip non-printable chars (<0x20 except tab/newline) from ADVERT
names in Go server decoder, Go ingestor decoder, and Node decoder.js.
#185: Add visual (N) badge next to node names when multiple nodes share
the same display name (case-insensitive). Shows in list, side pane, and
full detail page with 'also known as' links to other keys.
#186: Add packetsLast24h field to /api/stats response.
#187#188: Cache runtime.ReadMemStats() with 5s TTL in Go server.
#189: Temporarily patch HTMLCanvasElement.prototype.getContext during
L.heatLayer().addTo(map) to pass { willReadFrequently: true }, preventing
Chrome console warning about canvas readback performance.
Tests: 10 new tests for buildDupNameMap + dupNameBadge (143 total frontend).
Cache busters bumped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Kpa-clawbot