Kpa-clawbot 76d89e6578 fix(ingestor): exclude path_json='[]' rows from backfill WHERE (#1119) (#1121) ...

## Summary

`BackfillPathJSONAsync` re-selected observations whose `path_json` was
already `'[]'`, rewrote them to `'[]'`, and looped forever. The
`len(batch) == 0` exit condition was never reached, the migration marker
was never recorded, and the ingestor sustained 2–3 MB/s WAL writes at
idle (76% of CPU in `sqlite.Exec` per pprof).

## Fix

Drop `'[]'` from the WHERE clause:

```diff
WHERE o.raw_hex IS NOT NULL AND o.raw_hex != ''
- AND (o.path_json IS NULL OR o.path_json = '' OR o.path_json = '[]')
+ AND (o.path_json IS NULL OR o.path_json = '')
```

`'[]'` is the "already attempted, no hops" sentinel (still written at
line 994 of `cmd/ingestor/db.go` when `DecodePathFromRawHex` returns no
hops). Excluding it from the WHERE lets the loop terminate after one
full pass and the migration marker `backfill_path_json_from_raw_hex_v1`
to be recorded.

## TDD

- **Red commit** (`19f8004`):
`TestBackfillPathJSONAsync_BracketRowsTerminate` — seeds 100
observations with `path_json='[]'` and a `raw_hex` that decodes to zero
hops, asserts the migration marker is written within 5s. Fails on master
with *"backfill never recorded migration marker within 5s — infinite
loop on path_json='[]' rows"*.
- **Green commit** (`7019100`): WHERE-clause fix + updates
`TestBackfillPathJsonFromRawHex` row 1 expectation (the pre-seeded
`'[]'` row is now correctly skipped instead of being re-decoded).

## Test results

```
ok  	github.com/corescope/ingestor	49.656s
```

## Acceptance criteria from #1119

- [x] Backfill terminates within 1 polling cycle of having no progress
to make
- [x] Migration marker `backfill_path_json_from_raw_hex_v1` written
after termination
- [x] On restart, backfill recognizes migration done and exits
immediately (existing behavior — the migration check at the top of
`BackfillPathJSONAsync` was always correct; the bug was that the marker
never got written)
- [x] Test: seed DB with N observations all having `path_json = '[]'` →
backfill runs once → no UPDATEs issued, migration marker written
- [ ] Disk write rate on idle staging drops from 2–3 MB/s to <100 KB/s —
to be verified by the user post-deploy

Fixes #1119.

---------

Co-authored-by: OpenClaw Bot <bot@openclaw.local>

2026-05-05 17:35:16 -07:00

..

decrypt

feat: CLI tool to decrypt and export hashtag channel messages (#724)

2026-04-12 22:07:41 -07:00

ingestor

fix(ingestor): exclude path_json='[]' rows from backfill WHERE (#1119) (#1121)

2026-05-05 17:35:16 -07:00

server

fix(#662): GetRepeaterRelayInfo also looks up byPathHop by 1-byte prefix (#1086)

2026-05-05 02:33:27 -07:00