dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-10-06 12:17:20 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,967 Commits 83 Branches 38 Tags
f1edbbef3fd53db7c4ea0809937453932bdad514
Commit Graph
25 Commits
Author SHA1 Message Date
efitenSaarMesh-BotClaude Opus 5
 		40f664c587 chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941) 
Rebase of #1881 by @SaarMesh-Bot onto current master. Their three
commits are preserved, two of them cherry-picked with authorship intact;
the sweep itself had to be regenerated. Opened as a new PR rather than
force-pushing their branch.

Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2
landed as #1937.

## Why regenerated rather than merged

The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs
landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on
current master is cheaper and less error-prone than resolving 72
conflicts that are all whitespace. The drift it fixes also grew in the
meantime: 66 files now, against 72 then, but spread differently.

## The three commits

1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files.
2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet`
copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range
variable copied a `Config` embedding `sync.Once`. Cherry-picked
unchanged.
3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift
or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one
change, noted in the commit message: the ignore file pointed at
`04bc80ee`, the sweep commit on their branch, which does not exist on
this base and would make `git blame --ignore-revs-file` error. Repointed
at `d3a02599`, the sweep here.

## Verification

The claim "formatting only" is checked twice rather than asserted:

- Every changed file is byte-identical to `gofmt(previous content)`. 0
of 66 deviate.
- With line comments and all whitespace stripped, 0 of 66 files differ,
so no code outside comments changed.

14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt`
re-indents indented comment blocks to tabs and inserts a blank comment
line before them; the behavior matrix above `resolveHopWithContext` in
`cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own
output, not an edit, but it is worth naming because it makes the diff
look larger than "whitespace" suggests.

The gate was run locally exactly as the workflow runs it: `gofmt` clean,
and `go vet` clean in all 14 modules, including `cmd/ingestor` which is
what commit 2 fixes.

Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s),
`cmd/ingestor` passes except
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically
on bare master with "A required privilege is not held by the client"
(Windows symlink privilege on my host, not code).

## Sequencing

This should go last in the queue. The sweep touches 66 files, so merging
it before the remaining open Go PRs gives each of them a conflict about
nothing but formatting. After it lands the gate is active, and any PR
with drift fails CI until it runs `gofmt -w`.

Excluded from the sweep: the misnamed `Dockerfile.go`, which is a
Dockerfile that gofmt cannot parse (the workflow excludes it too), and
`docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a
spurious modification against `docs/deployment.md` and is unrelated.

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-03 18:52:03 +02:00
SaarMesh-BotSaarMesh-BotClaude
 		0d6f59ab2d fix(#1864): decode ANON_REQ source pubkey instead of treating it like REQUEST (#1866) 
Fixes #1864.

## Problem
`PAYLOAD_TYPE_ANON_REQ` was effectively treated like `REQUEST`. The two
differ on the wire:

```
REQUEST :  <dest hash 1B> <source hash 1B>          <hmac 2B> <encrypted>
ANON_REQ:  <dest hash 1B> <source pubkey 32B, full> <hmac 2B> <encrypted>
```

The decoders read the right bytes but surfaced the sender key as
`ephemeralPubKey`, which meant:
- `store.go`'s node indexer keys on `pubKey`/`destPubKey`/`srcPubKey`,
so ANON_REQ packets were **not** indexed — they didn't show up on a
node's packet view; and
- the packets list "details" rendered a bare `anon → <destHash>`,
throwing away the sender identity the packet actually carries.
- the detail side-view byte breakdown fell through the REQ catch-all,
mislabelling a nonexistent 1-byte "Src Hash" and placing
MAC/Encrypted-Data at the wrong offsets (`+2`/`+4` instead of
`+33`/`+35`).

## Fix
**Backend** (`cmd/ingestor` + `cmd/server` decoders)
- Surface the ANON_REQ sender key as `srcPubKey` (json) so it's indexed
and resolvable. The frontend keeps a legacy `ephemeralPubKey` reader so
packets decoded before this rename still resolve — no DB migration
needed.
- `TestDecodeAnonReqValid` now asserts the full 32-byte `srcPubKey`.

**Frontend**
- `hop-resolver.js`: new O(1) `nameForKey(pubkey)` using the existing
`pubkeyIdx` (all nodes).
- `getDetailPreview`: resolve the source pubkey to a node **name** when
known, else show the first 8 hex chars — no more bare `anon`.
- Detail side-view: explicit ANON_REQ breakdown — `Dest Hash (1B)` |
`Src Public Key (32B)` (node-linked) | `MAC @+33` | `Encrypted Data
@+35`.
- Detail header `srcLabel` falls back to the resolved ANON_REQ sender.
All rendered names are `escapeHtml`-wrapped.

## Testing
- `go test ./...` green for both `cmd/ingestor` and `cmd/server` (incl.
strengthened `TestDecodeAnonReqValid`).
- `node --check` on `packets.js`; brace/paren balance + markers verified
on `hop-resolver.js`.
- No HTML sink lines added → XSS preflight gate unaffected; every
interpolated name is escaped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-02 10:27:56 +02:00
Kpa-clawbotandclawbot b3189c613a fix(#1802): decode CONTROL DISCOVER_REQ/RESP subtype + body fields (#1806) 
## Summary
Extend CONTROL packet decoding to surface DISCOVER_REQ / DISCOVER_RESP
subtype plus body fields in the packet detail view. Previously only the
byte0 zero-hop flag was decoded; the body was rendered as opaque hex.

## What changed

**Backend** — `cmd/ingestor/decoder.go` `decodeControl()`
- New `Payload` fields (all omitempty): `CtrlSubtype`, `CtrlFilter`,
`CtrlTag`, `CtrlSince`, `CtrlNodeType`, `CtrlSNR`, `CtrlPubKey`.
- Subtype derived from `byte0 & 0xF0`: `0x80` → `DISCOVER_REQ`, `0x90` →
`DISCOVER_RESP`, otherwise `UNKNOWN`.
- REQ body parsed when `len(buf) >= 6`: `filter:u8 | tag:u32 LE`, plus
optional `since:u32 LE` when 4 more bytes remain.
- RESP body parsed when `len(buf) >= 6`: `node_type` (low nibble of
byte0), `snr:i8`, `tag:u32 LE`, and `pubkey` hex — 32 bytes when full, 8
bytes when prefix-only.
- Every field gated on length; short/truncated bodies emit subtype only
and never panic.
- `CtrlZeroHop` retained for backwards compatibility (rename flagged for
follow-up per triage).

**Frontend** — `public/packets.js` `getDetailPreview()`
- New `decoded.type === 'CONTROL'` branch renders subtype + present body
fields (filter / tag / since / node_type / snr / pubkey). Each field
shown only when populated, so truncated CONTROL still gets a subtype
label.

## Wire format reference
- `firmware/src/Mesh.cpp:69` — `CTL_TYPE_NODE_DISCOVER_REQ=0x80`,
`CTL_TYPE_NODE_DISCOVER_RESP=0x90`.
- `firmware/examples/simple_repeater/MyMesh.cpp:773-820` — body parse /
build.

## Tests (red → green, per AGENTS.md STRICT TDD)
- `cmd/ingestor/issue1802_test.go` — 6 cases: REQ full body (with
since), REQ no-since, RESP 32B pubkey, RESP 8B prefix pubkey, RESP
truncated pubkey (no panic, no pubkey emitted), short body (subtype
only), unknown subtype. Red commit `43713d3a` → green commit `d4b28180`.
Pre-existing CONTROL tests (`TestDecodeControlZeroHop`,
`TestDecodeControlMultiHop`) still pass.
- `test-packets.js` — 3 cases on `getDetailPreview`: DISCOVER_REQ
(filter+tag rendered), DISCOVER_RESP (snr+pubkey rendered), UNKNOWN
subtype label. Red commit `be23e349` → green commit `845d6c48`.

## Preflight overrides
- `check-branch-clean` (cross-stack): justified — issue #1802 explicitly
spans backend decoder (`cmd/ingestor/decoder.go`) and frontend renderer
(`public/packets.js`) per triage comment. Tests in both layers.
Single-purpose PR.

## Scope discipline
Files touched: `cmd/ingestor/decoder.go`,
`cmd/ingestor/issue1802_test.go`, `public/packets.js`,
`test-packets.js`. No other files. No firmware changes. No
`cmd/server/decoder.go` changes. No `CtrlZeroHop` rename (deferred per
triage).

Fixes #1802

---------

Co-authored-by: clawbot <bot@meshcore.local>
 2026-06-28 06:32:07 -07:00
Kpa-clawbotandcorescope-bot 9612f08e46 fix(#1610): decode firmware 1.16.0 extended ACK (5/6-byte payloads) (#1618) 
## Summary

Firmware 1.16.0 (`companion-v1.16.0`) ships variable-length
`PAYLOAD_TYPE_ACK` payloads: 4 bytes (legacy) → 5 bytes (4-byte CRC +
1-byte attempt, commit `f6e6fdaa`) → 6 bytes (+ 1-byte RNG, commit
`a130a95a`). CoreScope's decoder previously truncated past the 4-byte
CRC and discarded the attempt + RNG bytes.

This PR teaches `cmd/ingestor/decoder.go` to surface the extended bytes
on the decoded payload so the DB/UI can distinguish v1.15 vs v1.16
senders, with no schema or wire-compat changes.

Partial fix for #1610 — top-level ACK + multipart-inner ACK are covered.
PATH-extra ACK parsing (`decodePathPayload`) is deferred to #1612 per
triage.

## Changes

- `decodeAck` reads 4/5/6-byte payloads. Keeps `extraHash` (4-byte CRC)
for compat; adds optional `ackLen`, `ackAttempt`, `ackRand` JSON fields.
Legacy 4-byte ACKs leave attempt/rand `nil`.
- `decodeMultipart` ACK branch relaxes the `len >= 5` floor so the inner
blob can be 4/5/6 bytes (multipart `payload_len` 5/6/7). Adds
`innerAckLen`, `innerAckAttempt`, `innerAckRand`.
- All additions are `omitempty` — backwards-compatible JSON only. No DB
column, no schema migration, no frontend change.

## Out of scope (per issue triage)

- `decodePathPayload` PATH-extra parsing — tracked separately in #1612.
- Frontend rendering of attempt counter — leave for a follow-up if the
DB/UI eventually wants to display it.

## TDD

- **Red commit `3fce0465`** adds `cmd/ingestor/issue1610_test.go` with 6
new assertions (legacy 4-byte, extended 5/6-byte, multipart variants of
each). New fields are declared on `Payload` so the test compiles, but no
decoder populates them yet — tests fail on `ackLen=<nil> want 4` etc.
Verified isolation with `git stash` of decoder.go + re-run.
- **Green commit `5165c202`** implements the decoder changes. `go test
./...` in `cmd/ingestor` passes.

## Fixtures

Synthetic wire vectors built by hand against the firmware spec — the
issue did not provide real captures. Each test cites the firmware ref +
commit it derives from (`BaseChatMesh.cpp:218-234`, commits `f6e6fdaa`
and `a130a95a`).

## References

- Issue #1610
- Firmware tag `companion-v1.16.0` @ `07a3ca9e`
- Upstream PR meshcore-dev/MeshCore#2594
- Blog: https://blog.meshcore.io/2026/06/06/release-1-16-0

---------

Co-authored-by: corescope-bot <bot@corescope.local>
 2026-06-06 21:05:50 -07:00
Joel Claw 95d7916530 fix(channels): normalize known channel display names (public → Public) (#777) 
Normalizes well-known channel display names (currently only `public` → `Public`) so existing deployments with pre-#761 lowercase config keys show the canonical firmware-default name `Public` in the UI.

Behavior:
- `knownChannelCasing` lookup (`decoder.go`) — single-entry map, easy to extend.
- `normalizeChannelName()` applied at config load (`loadChannelKeys`) AND at decode time (defense in depth).
- One-shot SQLite migration `channel_hash_casing_v1` backfills `channel_hash='public'` → `'Public'` on `payload_type=5` rows so channel-grouping queries don't split across the upgrade boundary.
- Hardcoded list intentionally tiny (1 entry); custom/user channels left untouched.

Safety:
- Channel-hash derivation (`SHA256(channelName)[:16]` for `#`-prefixed `HashChannels`) is unchanged — normalization only renames map keys for explicit `ChannelKeys` entries (which don't feed `deriveHashtagChannelKey`).
- PSK lookup is by hash byte, not by name — mesh interop preserved.
- Migration is gated by `_migrations.name='channel_hash_casing_v1'`, idempotent.

Tests (`cmd/ingestor/normalize_channel_test.go`):
- `TestNormalizeChannelName` covers known + hashtag + custom + empty.
- `TestLoadChannelKeys_NormalizesKnownDisplayNames` — verifies `public` → `Public` at load.
- `TestLoadChannelKeys_LeavesCustomNamesUntouched` — custom names not auto-capitalized.
- `TestLoadChannelKeys_DuplicateCasingLogsWarning` — config containing both casings resolves deterministically (canonical wins).

Mutation test confirmed: reverting load-time normalize → `TestLoadChannelKeys_NormalizesKnownDisplayNames` and `_DuplicateCasingLogsWarning` both fail on assertions.

Related: #761
 2026-05-25 23:05:07 -07:00
efitenClaude Sonnet 4.6Kpa-clawbotopenclaw-bot
 		2329639f45 feat: scoped/unscoped transport-route statistics (#899) (#915) 
@
## What this PR does

Implements region-scoped transport-route packet tracking with two
sub-features:

### Feature 1 — Scope statistics (`scope_name`)
- At ingest, transport-route packets (route_type 0/3) with Code1 !=
`0000` are HMAC-matched against configured `hashRegions` keys (mirroring
the `hashChannels` pattern). Matched region name (or `""` for unknown)
stored in new `transmissions.scope_name` column via migration
`scope_name_v1`.
- New `GET /api/scope-stats?window=` endpoint (1h/24h/7d, 30s
server-side TTL) returning transport totals, scoped/unscoped counts,
per-region breakdown, and time-series.
- New **Scopes** tab in Analytics with summary cards, per-region table,
and two-line SVG chart. Auto-refreshes every 60s.

### Feature 2 — Node default scope (`default_scope`)
- Per-node `default_scope` column on `nodes`/`inactive_nodes` (migration
`nodes_default_scope_v1`) tracks the most recently matched region for
each node, derived from transport-scoped ADVERT packets.
- `GET /api/nodes` response includes `default_scope` field when column
is present.
- Node detail panel displays the default scope badge.
- Async startup backfill (`BackfillDefaultScopeAsync`) populates the
column for nodes with pre-existing ADVERT data.

### Config
Add `hashRegions` to `config.json` (see `config.example.json`). One
entry per region name (with or without leading `#`).
@

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
 2026-05-21 14:00:06 -07:00
Kpa-clawbotandbot 749fdc114f feat(decoder+ui): close remaining P2 items from #1279 — payloadTypeNames, legend, TransportCodes, Feat1/2, RAW_CUSTOM, sensor docs (#1291) 
RED commit: `dc4c0800` — CI:
https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1279-p2

Closes the remaining six 🟢 P2 items in umbrella #1279 (PR #1280 shipped
P0+P1, PR #1276 shipped ACK/RESPONSE/PATH legend rows).

### Item-by-item

| # | Item | Where | Test |
|---|---|---|---|
| 1 | `payloadTypeNames` parity | `cmd/server/store.go` |
`cmd/server/issue1279_p2_test.go::TestPayloadTypeNamesAll13` |
| 2 | Legend rows: Anon Req / Grp Data / Multipart / Control / Raw
Custom | `public/live.js` | `test-issue-1279-legend-p2-e2e.js`
(Playwright) |
| 3 | TransportCodes detail-row + `code1=` / `code2=` filter grammar |
`public/packets.js`, `public/packet-filter.js` |
`test-issue-1279-p2-code-filter.js` (6 cases) |
| 4 | Multibyte capability badge on node detail/list rows |
`public/nodes.js::renderNodeBadges` | `n.hash_size >= 2` (observable
Feat1/Feat2 proxy; firmware `AdvertDataHelpers.h:14-16`) |
| 5 | RAW_CUSTOM (0x0F) `{rawLength, firstByteTag}` decode + detail-row
| `cmd/server/decoder.go`, `cmd/ingestor/decoder.go`,
`public/packets.js` | `TestDecodeRawCustomExposesLengthAndTag` × 2 +
updated `TestDecodePayloadRAWCustom` |
| 6 | Sensor advert telemetry firmware-derivation comments |
`cmd/ingestor/decoder.go:363-380` | pure comments — exempt per AGENTS |

### Firmware refs cited inline
- `firmware/src/Packet.h:19-32` — PAYLOAD_TYPE_* constants
- `firmware/src/Packet.h:46` — TransportCodes wire layout
- `firmware/src/Mesh.cpp:577` — `createRawData`
- `firmware/src/helpers/SensorMesh.{h,cpp}` — sensor advert telemetry
derivation
- `firmware/src/helpers/AdvertDataHelpers.h:14-16` — Feat1/Feat2

### TDD
Red `dc4c0800` proves the assertions gate behavior:
- `payloadTypeNames` had only 12 entries (no 0x0F).
- RAW_CUSTOM decoded as `UNKNOWN` with no envelope fields.

Green `<HEAD>` makes both green; per-item tests included.

### Cross-stack note
Cross-stack: justified — items 1/5 add decoder output fields; items
2/3/4/5 surface those fields in the UI in the same PR per #1279
acceptance.

### Out of scope
Item 4 surfaces the observable multibyte capability via the persisted
`hash_size` (Feat1/Feat2 wire bits are only on transient adverts and not
stored per-node today); persisting raw Feat1/Feat2 per-node is left for
a follow-up.

Fixes #1279

---------

Co-authored-by: bot <bot@corescope>
 2026-05-19 08:08:28 -07:00
Kpa-clawbotandOpenClaw Bot e6c30e1a7e feat(decoder): GRP_DATA + MULTIPART + advertRole fix + CONTROL flags (#1279 P0+P1) (#1280) 
Addresses the four P0+P1 firmware reconciliation gaps from the umbrella
audit (issue #1279). RED commit: `0a4c084e` (asserts on stub returns;
all 13 assertions fail). GREEN commit: `13867681`.

## What's in this PR

### P0 — silently dropped data

- **#1 GRP_DATA (0x06) decoder.** Outer envelope is the same shape as
GRP_TXT (`channel_hash(1)+MAC(2)+ciphertext`) per
`firmware/src/helpers/BaseChatMesh.cpp:476,500`. Factored
`decryptChannelBlock(...)` helper used by both 5 and 6. When a channel
key matches, the inner is parsed per
`firmware/src/helpers/BaseChatMesh.cpp:382-385` as `data_type(uint16 LE)
+ data_len(1) + blob(data_len)`. Surfaces `{channelHash, MAC, dataType,
dataLen, decryptedBlob}` on decrypt or `{channelHash, MAC,
encryptedData}` otherwise. Server-side decoder surfaces envelope only
(no key store).
- **#2 MULTIPART (0x0A) decoder.** Per `firmware/src/Mesh.cpp:289`,
byte0 = `(remaining<<4) | inner_type`. When `inner_type ==
PAYLOAD_TYPE_ACK (0x03)`, next 4 bytes are the LE ack_crc per
`firmware/src/Mesh.cpp:292-307`. Surfaces `{remaining, innerType,
innerTypeName, innerAckCrc | innerPayload}`.

### P1 — mis-classified / opaque

- **#3 `advertRole()` raw-type fix.** Per
`firmware/src/helpers/AdvertDataHelpers.h:7-12`, ADV_TYPE_NONE = 0 and
5-15 are FUTURE. The previous boolean fallback collapsed both into
`"companion"`, silently relabelling unknown/reserved types. New
behaviour: type 0 → `none`, 1 → `companion`, 2-4 →
`repeater`/`room`/`sensor`, 5-15 → `type-N`. `ValidateAdvert` accepts
the new labels.
- **#4 CONTROL (0x0B) byte0 flags + length.** Per
`firmware/src/Mesh.cpp:69` + `createControlData` at `Mesh.cpp:609`,
byte0 high-bit marks the zero-hop direct subset. Surfaces `{ctrlFlags,
ctrlZeroHop, ctrlLength}`.

### Drift fix

- `cmd/server/store.go` `payloadTypeNames` now includes `6: GRP_DATA`
and `10: MULTIPART` (previously omitted; canonical decoder map already
had them).

## Lockstep & TDD

Both `cmd/ingestor/decoder.go` and `cmd/server/decoder.go` updated in
the same commits — same wire-vector tests live in both packages
(`cmd/{ingestor,server}/issue1279_test.go`). Per-item RED→GREEN visible
in `git log`.

| Item | Tests | RED proof |
|---|---|---|
| #1 GRP_DATA | ingestor: NoKey + DecryptedInner; server: Envelope | 6
assertions failed pre-impl |
| #2 MULTIPART | ingestor + server: Ack + NonAck | 8 assertions failed
pre-impl |
| #3 advertRole | ingestor + server: 7-row table | 3 assertions failed
pre-impl |
| #4 CONTROL | ingestor + server: ZeroHop + MultiHop | 6 assertions
failed pre-impl |

## What's NOT in this PR

The umbrella issue lists P2 items that ship in follow-up PRs:

- Live + compare legend entries for the long tail of newly-named types
(#1274 + others).
- TransportCodes UI surface + filter grammar.
- feat1/feat2 capability badges.
- `payloadTypeNames` consolidation across server/ingestor
(drift-prevention).

Leave the umbrella open after this merges.

Refs #1279

---------

Co-authored-by: OpenClaw Bot <bot@openclaw.local>
 2026-05-18 23:19:27 -07:00
Kpa-clawbotcorescope-botopenclaw-botcorescope-bot
 		85e97d2f37 fix(#1211): bounds-check path length to prevent slice [218:15] panic in MQTT decode (#1214) 
**RED commit:** `65d9f57b` (CI run will appear at
https://github.com/Kpa-clawbot/CoreScope/actions after PR opens)

Fixes #1211

## Root cause

`decodePath()` returns `bytesConsumed = hash_size * hash_count` where
both come straight from the wire-supplied `pathByte` (upper 2 bits →
`hash_size`, lower 6 bits → `hash_count`). Max claimable: 4 × 63 = 252
bytes.

A malformed packet on the wire claimed `pathByte=0xF6` (hash_size=4,
hash_count=54 → 216 path bytes) inside a 15-byte buffer. The inner
hop-extraction loop in `decodePath` did break early on overflow — but
`bytesConsumed` was still returned at face value (216). `DecodePacket`
then did `offset += 216` (offset=218) and `payloadBuf := buf[offset:]`
panicked with the prod-observed signature:

```
runtime error: slice bounds out of range [218:15]
```

The handler-level `defer/recover` at `cmd/ingestor/main.go:258-263`
caught it, but the message was silently dropped with no usable
diagnostic.

## Fix

Add a `if offset > len(buf)` guard at BOTH decoder sites (same pattern,
same panic potential):

- `cmd/ingestor/decoder.go` — DecodePacket after decodePath
- `cmd/server/decoder.go` — DecodePacket after decodePath

Return a descriptive error citing the claimed length and pathByte hex so
operators can reproduce.

Also: `cmd/ingestor/main.go` decode-error log now includes `topic`,
`observer`, and `rawHexLen` so future malformed packets are reproducible
without needing to attach a debugger.

## Tests (TDD red → green)

Both packages got two new tests:

- **`TestDecodePacketBoundsFromWire_Issue1211`** — feeds the exact wire
shape from the prod log (`pathByte=0xF6` inside a 15-byte buf). Asserts
`DecodePacket` does NOT panic and returns an error.
- **`TestDecodePacketFuzzTruncated_Issue1211`** — sweeps every `(header,
pathByte)` combination with tails 0..19 bytes (≈1.3M inputs). Asserts
zero panics.

### Red commit proof

On commit `65d9f57b` (RED), both tests fail with the panic:
```
=== RUN   TestDecodePacketBoundsFromWire_Issue1211
    decoder_test.go:1996: DecodePacket panicked on malformed input: runtime error: slice bounds out of range [218:15]
--- FAIL: TestDecodePacketBoundsFromWire_Issue1211 (0.00s)
=== RUN   TestDecodePacketFuzzTruncated_Issue1211
    decoder_test.go:2010: DecodePacket panicked during fuzz: runtime error: slice bounds out of range [3:2]
--- FAIL: TestDecodePacketFuzzTruncated_Issue1211 (0.01s)
```

On commit `7a6ae52c` (GREEN), full suites pass:
- `cmd/ingestor`: `ok 53.988s`
- `cmd/server`:   `ok 29.456s`

## Acceptance criteria

- [x] Identify the slice op producing `[218:15]` — `payloadBuf :=
buf[offset:]` in `DecodePacket` (decoder.go), where `offset` had been
advanced by an unchecked `bytesConsumed` from `decodePath()`.
- [x] Bounds check added at the identified site(s) — both ingestor and
server decoders.
- [x] Test with crafted payload (length-field > remaining buffer) —
`TestDecodePacketBoundsFromWire_Issue1211`.
- [x] Log topic, observer ID, payload byte length on drop — updated
`MQTT [%s] decode error` log line.
- [x] Existing tests stay green — confirmed both packages.

## Out of scope

Reconnect-after-disconnect (#1212) — handled by a separate subagent.
This PR touches NO reconnect logic.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: corescope-bot <bot@corescope>
 2026-05-15 22:34:21 -07:00
Kpa-clawbotandcorescope-bot c9301fee9c fix(ingestor): extract per-hop SNR for TRACE packets at ingest time (#1028) 
## Problem

PR #1007 added per-hop SNR extraction (`snrValues`) for TRACE packets to
`cmd/server/decoder.go`. That code path is only hit by the on-demand
re-decode endpoint (packet detail). The actual ingest pipeline runs
`cmd/ingestor/decoder.go`, decodes the packet once, and persists
`decoded_json` into SQLite. The server then serves `decoded_json` as-is
for list/feed queries.

Net effect: `snrValues` never appears in any production response,
because the ingestor's decoder was never updated.

Confirmed empirically: `strings /app/corescope-ingestor | grep snrVal`
returns nothing.

## Fix

Port the SNR extraction logic from `cmd/server/decoder.go` (lines
410–422) into `cmd/ingestor/decoder.go`. For TRACE packets, the header
path bytes are int8 SNR values in quarter-dB encoding; extract them into
`payload.SNRValues` **before** `path.Hops` is overwritten with
payload-derived hop IDs.

Also adds the matching `SNRValues []float64` field to the ingestor's
`Payload` struct so it serializes into `decoded_json`.

## TDD

- **Red commit** (`6ae4c07`): adds `TestDecodeTraceExtractsSNRValues` +
`SNRValues` field stub. Compiles, fails on assertion (`len(SNRValues)=0,
want 2`).
- **Green commit** (`4a4f3f3`): adds extraction loop. Test passes.

Test packet: `26022FF8116A23A80000000001C0DE1000DEDE`
- header `0x26` = TRACE + DIRECT
- pathByte `0x02` = hash_size 1, hash_count 2
- header path `2F F8` → SNR `[int8(0x2F)/4, int8(0xF8)/4]` = `[11.75,
-2.0]`

## Files

- `cmd/ingestor/decoder.go` — `+16` (field + extraction)
- `cmd/ingestor/decoder_test.go` — `+29` (red test)

## Out of scope

- `cmd/server/decoder.go` is already correct (PR #1007). Untouched.
- Backfill of historical `decoded_json` rows. New TRACE packets get SNR;
old rows do not until re-decoded.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
 2026-05-03 21:42:14 -07:00
Kpa-clawbotandyou d43c95a4bb fix(ingestor): warn when TRACE payload decode fails but observation stored (closes #889) (#992) 
## Summary

Closes #889.

When a TRACE packet's payload is too short to decode (< 9 bytes),
`decodeTrace` returns an error in `Payload.Error` but the observation is
still stored with empty `Path.Hops`. Previously this was completely
silent — no log, no anomaly flag, no indication the row is degraded.

This fix populates `DecodedPacket.Anomaly` with the decode error message
(e.g., `"TRACE payload decode failed: too short"`) so operators and
downstream consumers can identify degraded observations.

## TDD Commit History

1. **Red commit** `04e0165` — failing test asserting `Anomaly` is set
when TRACE payload decode fails
2. **Green commit** `d3e72d1` — 3-line fix in `decoder.go` line 601-603:
check `payload.Error != ""` for TRACE packets and set anomaly

## What Changed

`cmd/ingestor/decoder.go` (lines 601-603): Added a check before the
existing TRACE path-parsing block. If `payload.Error` is non-empty for a
TRACE packet, `anomaly` is set to `"TRACE payload decode failed:
<error>"`.

`cmd/ingestor/decoder_test.go`: Added
`TestDecodeTracePayloadFailSetsAnomaly` — constructs a TRACE packet with
a 4-byte payload (too short), asserts the packet is still returned
(observation stored) and `Anomaly` is populated.

## Verification

- `go build ./...` ✓
- `go test ./...` ✓ (all pass including new test)
- Anti-tautology: reverting the fix causes the new test to fail (asserts
`pkt.Anomaly == ""` → error)

---------

Co-authored-by: you <you@example.com>
 2026-05-02 20:34:27 -07:00
Kpa-clawbotandyou 56ec590bc4 fix(#886): derive path_json from raw_hex at ingest (#887) 
## Problem

Per-observation `path_json` disagrees with `raw_hex` path section for
TRACE packets.

**Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡`
- `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE
payload)
- `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header)

## Root Cause

`DecodePacket` correctly parses TRACE packets by replacing `path.Hops`
with hop IDs from the payload's `pathData` field (the actual route).
However, the header path bytes for TRACE packets contain **SNR values**
(one per completed hop), not hop IDs.

`BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which
for TRACE packets contained the payload-derived hops — not the header
path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex`
to describe completely different paths.

## Fix

- Added `DecodePathFromRawHex(rawHex)` — extracts header path hops
directly from raw hex bytes, independent of any TRACE payload
overwriting.
- `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of
using `decoded.Path.Hops`, guaranteeing `path_json` always matches the
`raw_hex` path section.

## Tests (8 new)

**`DecodePathFromRawHex` unit tests:**
- hash_size 1, 2, 3, 4
- zero-hop direct packets
- transport route (4-byte transport codes before path)

**`BuildPacketData` integration tests:**
- TRACE packet: asserts path_json matches raw_hex header path (not
payload hops)
- Non-TRACE packet: asserts path_json matches raw_hex header path

All existing tests continue to pass (`go test ./...` for both ingestor
and server).

Fixes #886

---------

Co-authored-by: you <you@example.com>
 2026-04-21 21:13:58 -07:00
Kpa-clawbotandyou a8e1cea683 fix: use payload type bits only in content hash (not full header byte) (#787) 
## Problem

The firmware computes packet content hash as:

```
SHA256(payload_type_byte + [path_len for TRACE] + payload)
```

Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type
bits (2-5).

CoreScope was using the **full header byte** in its hash computation,
which includes route type bits (0-1) and version bits (6-7). This meant
the same logical packet produced different content hashes depending on
route type — breaking dedup and packet lookup.

**Firmware reference:** `Packet.cpp::calculatePacketHash()` uses
`getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) &
PH_TYPE_MASK`.

## Fix

- Extract only payload type bits: `payloadType := (headerByte >> 2) &
0x0F`
- Include `path_len` byte in hash for TRACE packets (matching firmware
behavior)
- Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go`

## Tests Added

- **Route type independence:** Same payload with FLOOD vs DIRECT route
types produces identical hash
- **TRACE path_len inclusion:** TRACE packets with different `path_len`
produce different hashes
- **Firmware compatibility:** Hash output matches manual computation of
firmware algorithm

## Migration Impact

Existing packets in the DB have content hashes computed with the old
(incorrect) formula. Options:

1. **Recompute hashes** via migration (recommended for clean state)
2. **Dual lookup** — check both old and new hash on queries (backward
compat)
3. **Accept the break** — old hashes become stale, new packets get
correct hashes

Recommend option 1 (migration) as a follow-up. The volume of affected
packets depends on how many distinct route types were seen for the same
logical packet.

Fixes #786

---------

Co-authored-by: you <you@example.com>
 2026-04-18 11:52:22 -07:00
Kpa-clawbotandyou 14367488e2 fix: TRACE path_json uses path_sz from flags byte, not header hash_size (#732) 
## Summary

TRACE packets encode their route hash size in the flags byte (`flags &
0x03`), not the header path byte. The decoder was using `path.HashSize`
from the header, which could be wrong or zero for direct-route TRACEs,
producing incorrect hop counts in `path_json`.

## Protocol Note

Per firmware, TRACE packets are **always direct-routed** (route_type 2 =
DIRECT, or 3 = TRANSPORT_DIRECT). FLOOD-routed TRACEs (route_type 1) are
anomalous — firmware explicitly rejects TRACE via flood. The decoder
handles these gracefully without crashing.

## Changes

**`cmd/server/decoder.go` and `cmd/ingestor/decoder.go`:**
- Read `pathSz` from TRACE flags byte: `(traceFlags & 0x03) + 1`
(0→1byte, 1→2byte, 2→3byte)
- Use `pathSz` instead of `path.HashSize` for splitting TRACE payload
path data into hops
- Update `path.HashSize` to reflect the actual TRACE path size
- Added `HopsCompleted` field to ingestor `Path` struct for parity with
server
- Updated comments to clarify TRACE is always direct-routed per firmware

**`cmd/server/decoder_test.go` — 5 new tests:**
- `TraceFlags1_TwoBytePathSz`: flags=1 → 2-byte hashes via DIRECT route
- `TraceFlags2_ThreeBytePathSz`: flags=2 → 3-byte hashes via DIRECT
route
- `TracePathSzUnevenPayload`: payload not evenly divisible by path_sz
- `TraceTransportDirect`: route_type=3 with transport codes + TRACE path
parsing
- `TraceFloodRouteGraceful`: anomalous FLOOD+TRACE handled without crash

All existing TRACE tests (flags=0, 1-byte hashes) continue to pass.

Fixes #731

---------

Co-authored-by: you <you@example.com>
 2026-04-13 08:20:09 -07:00
copelajeandyou 922ebe54e7 BYOP Advert signature validation (#686) 
For BYOP mode in the packet analyzer, perform signature validation on
advert packets and display whether successful or not. This is added as
we observed many corrupted advert packets that would be easily
detectable as such if signature validation checks were performed.

At present this MR is just to add this status in BYOP mode so there is
minimal impact to the application and no performance penalty for having
to perform these checks on all packets. Moving forward it probably makes
sense to do these checks on all advert packets so that corrupt packets
can be ignored in several contexts (like node lists for example).

Let me know what you think and I can adjust as needed.

---------

Co-authored-by: you <you@example.com>
 2026-04-12 04:02:17 +00:00
efitenClaude Sonnet 4.6you
 		144e98bcdf fix: hide hash size for zero-hop direct adverts (#649) (#653) 
## Fix: Zero-hop DIRECT packets report bogus hash_size

Closes #649

### Problem
When a DIRECT packet has zero hops (pathByte lower 6 bits = 0), the
generic `hash_size = (pathByte >> 6) + 1` formula produces a bogus value
(1-4) instead of 0/unknown. This causes incorrect hash size displays and
analytics for zero-hop direct adverts.

### Solution

**Frontend (JS):**
- `packets.js` and `nodes.js` now check `(pathByte & 0x3F) === 0` to
detect zero-hop packets and suppress bogus hash_size display.

**Backend (Go):**
- Both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` reset
`HashSize=0` for DIRECT packets where `pathByte & 0x3F == 0` (hash_count
is zero).
- TRACE packets are excluded since they use hashSize to parse hop data
from the payload.
- The condition uses `pathByte & 0x3F == 0` (not `pathByte == 0x00`) to
correctly handle the case where hash_size bits are non-zero but
hash_count is zero — matching the JS frontend approach.

### Testing

**Backend:**
- Added 4 tests each in `cmd/server/decoder_test.go` and
`cmd/ingestor/decoder_test.go`:
  - DIRECT + pathByte 0x00 → HashSize=0 ✅
- DIRECT + pathByte 0x40 (hash_size bits set, hash_count=0) → HashSize=0
✅
  - Non-DIRECT + pathByte 0x00 → HashSize=1 (unchanged) ✅
  - DIRECT + pathByte 0x01 (1 hop) → HashSize=1 (unchanged) ✅
- All existing tests pass (`go test ./...` in both cmd/server and
cmd/ingestor)

**Frontend:**
- Verified hash size display is suppressed for zero-hop direct adverts

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
 2026-04-07 19:39:15 -07:00
Kpa-clawbot 5aa4fbb600 chore: normalize all files to LF line endings 2026-03-30 22:52:46 -07:00
efitenandClaude Sonnet 4.6 8f833f64ae fix: parse TRACE packet path hops from payload instead of header (#277) 
Fixes #276

## Root cause

TRACE packets store hop IDs in the payload (bytes 9+) rather than in the
header path field. The header path field is overloaded in TRACE packets
to carry RSSI values instead of repeater IDs (as noted in the issue
comments). This meant `Path.Hops` was always empty for TRACE packets —
the raw bytes ended up as an opaque `PathData` hex string with no
structure.

The hashSize encoded in the header path byte (bits 6–7) is still valid
for TRACE and is used to split the payload path bytes into individual
hop prefixes.

## Fix

After decoding a TRACE payload, if `PathData` is non-empty, parse it
into individual hops using `path.HashSize`:

```go
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
    pathBytes, err := hex.DecodeString(payload.PathData)
    if err == nil && path.HashSize > 0 {
        for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
            path.Hops = append(path.Hops, ...)
        }
    }
}
```

Applied to both `cmd/ingestor/decoder.go` and `cmd/server/decoder.go`.

## Verification

Packet from the issue: `260001807dca00000000007d547d`

| | Before | After |
|---|---|---|
| `Path.Hops` | `[]` | `["7D", "54", "7D"]` |
| `Path.HashCount` | `0` | `3` |

New test `TestDecodeTracePathParsing` covers this exact packet.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-30 16:27:50 +00:00
you 37396823ad fix: align Go packet decoder with MeshCore firmware spec 
Match the C++ firmware wire format (Packet::writeTo/readFrom):

1. Field order: transport codes are parsed BEFORE path_length byte,
   matching firmware's header → transport_codes → path_len → path → payload

2. ACK payload: just 4-byte CRC checksum, not dest+src+ackHash.
   Firmware createAck() writes only ack_crc (4 bytes).

3. TRACE payload: tag(4) + authCode(4) + flags(1) + pathData,
   matching firmware createTrace() and onRecvPacket() TRACE handler.

4. ADVERT features: parse feat1 (0x20) and feat2 (0x40) optional
   2-byte fields between location and name, matching AdvertDataBuilder
   and AdvertDataParser in the firmware.

5. Transport code naming: code1/code2 instead of nextHop/lastHop,
   matching firmware's transport_codes[0]/transport_codes[1] naming.

Fixes applied to both cmd/ingestor/decoder.go and cmd/server/decoder.go.
Tests updated to match new behavior.
 2026-03-29 07:50:51 -07:00
copilot-swe-agent[bot]andKpaBap a827fd3b43 fix: gate telemetry on sensor flag, fix 0°C emission, safe migration with PRAGMA check 
Agent-Logs-Url: https://github.com/Kpa-clawbot/meshcore-analyzer/sessions/1c2af64b-0e8a-4dd0-ae80-e296f70437e9

Co-authored-by: KpaBap <746025+KpaBap@users.noreply.github.com>
 2026-03-28 20:35:50 +00:00
Kpa-clawbotandCopilot 54cbc648e0 feat: decode telemetry from adverts — battery voltage + temperature on nodes 
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.

Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
  (battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
  to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
  DB migration test, node telemetry update test, server API shape tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-28 12:07:42 -07:00
Kpa-clawbotandCopilot bcf7159538 fix: detect garbage text after channel decryption, fixes #197 
After decryption produces text, validate it's printable UTF-8.
If it contains more than 2 non-printable characters (excluding
newline/tab), mark as decryption_failed with text: null.

Applied to both Node (decoder.js) and Go (cmd/ingestor/decoder.go)
decoders. Added tests for garbage and valid text in both.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 21:48:37 -07:00
Kpa-clawbotandCopilot 77988ded3e fix: #184-#189 — sanitize names, packetsLast24h, ReadMemStats cache, dup name indicator, heatmap warning 
#184: Strip non-printable chars (<0x20 except tab/newline) from ADVERT
names in Go server decoder, Go ingestor decoder, and Node decoder.js.

#185: Add visual (N) badge next to node names when multiple nodes share
the same display name (case-insensitive). Shows in list, side pane, and
full detail page with 'also known as' links to other keys.

#186: Add packetsLast24h field to /api/stats response.

#187 #188: Cache runtime.ReadMemStats() with 5s TTL in Go server.

#189: Temporarily patch HTMLCanvasElement.prototype.getContext during
L.heatLayer().addTo(map) to pass { willReadFrequently: true }, preventing
Chrome console warning about canvas readback performance.

Tests: 10 new tests for buildDupNameMap + dupNameBadge (143 total frontend).
Cache busters bumped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 20:50:08 -07:00
Kpa-clawbotandCopilot 91a8c0405f feat(go): implement channel decryption for GRP_TXT packets, fixes #176 
Go ingestor never had channel decryption — GRP_TXT packets were stored
with raw encrypted data while Node.js decoded them successfully.

Changes:
- decoder.go: Add decryptChannelMessage() implementing MeshCore channel
  crypto (HMAC-SHA256 MAC verification + AES-128-ECB decryption), matching
  the algorithm in @michaelhart/meshcore-decoder. Update decodeGrpTxt(),
  decodePayload(), and DecodePacket() to accept and pass channel keys.
  Add Payload fields: ChannelHashHex, DecryptionStatus, Channel, Text,
  Sender, SenderTimestamp.
- config.go: Add ChannelKeysPath and ChannelKeys fields to Config struct.
- main.go: Add loadChannelKeys() that loads channel-rainbow.json (same
  file used by Node.js server) from beside the config file, with env var
  and config overrides. Pass loaded keys through the decoder pipeline.
- decoder_test.go: Add 14 channel decryption tests covering valid
  decryption, MAC failure, wrong key, no-sender messages, bracket
  sender exclusion, key iteration, channelHashHex formatting, and
  decryption status states. Cross-validated against Node.js output.
- Update all DecodePacket/decodePayload/decodeGrpTxt/handleMessage call
  sites in test files to pass the new channelKeys parameter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 17:55:52 -07:00
Kpa-clawbotandCopilot 488ead617d feat: add standalone Go MQTT ingestor (cmd/ingestor/) 
First step of Go rewrite — separates MQTT ingestion from the Node.js
web server. Single static binary (no CGO) that connects to MQTT
brokers, decodes MeshCore packets, and writes to the shared SQLite DB.

Ported from JS:
- decoder.js → decoder.go (header, path, all payload types, adverts)
- computeContentHash → Go (SHA-256, path-independent)
- db.js v3 schema → db.go (transmissions, observations, nodes, observers)
- server.js MQTT logic → main.go (multi-broker, reconnect, IATA filter)

25 Go tests passing (golden fixtures from production + schema compat).
No existing JS files modified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-26 23:22:26 -07:00