Files
meshcore-analyzer/cmd/server/hash_migration_status_test.go
ZachandClaude Opus 5.5 7ae6645337 fix(#1856): run the content-hash migration in the ingestor (#2409)
## Summary

Fixes #1856. The content-hash migration from #787 has done nothing since
#1283. It ran in the server on the read-only analyzer handle, so every
batch failed. Before #1958 it also reported `hashMigrationComplete:
true` anyway. It also meant the server issued `UPDATE` and `DELETE`
against a database it must never write.

The migration now runs in the ingestor as the async migration
`content_hash_payload_bits_v1`. Split out of #2396 at review's request.

## Who is affected

Only installs holding transmissions from **before #787 (April 18)** that
did not run a server between #787 and #1283 (April 18 to May 18), the
only month the old server-side pass could commit. `retention.packetDays`
defaults to keep-forever, so long-lived installs can qualify. For
everyone else this is a one-time scan that changes nothing.

## What changes

- Rows present at boot are rescanned by id in batches of 5000, read
outside the writer lock. Stale rows commit 250 per transaction (the
retention bound), so ingest waits for one chunk at most, not a whole
batch.
- After each scan batch that changed rows, a progress line is logged.
- A stale hash is rewritten in place.
- A row whose new hash already exists is **merged** into that survivor:
- its observations move, except any that would violate the dedup index;
  - the survivor's `first_seen` and `last_seen` widen to cover both;
- the duplicate is deleted with the same child cleanup retention uses
(now one shared helper).
- Collisions are found by lookup, not by catching a constraint error.
- The server copy is deleted. `/api/stats.hashMigrationComplete` keeps
its shape and now reports the ingestor's `_async_migrations` status.
- `readonly_invariant_test.go` now fails on any `Begin` or
`INSERT`/`UPDATE`/`DELETE` against the analyzer DB from `cmd/server`.

## Known limitation: a running server does not follow the pass

A server that loaded these rows before the pass keeps the old hashes,
the merged duplicates, and survivors without their moved observations
until it restarts. That's deliberate:
- `tx_rewrite_feed` cannot carry a merge. Its rows cascade away with the
deleted duplicate, and the server's tail refreshes only `decoded_json`
and `scope_name`.
- Removing a transmission from every in-memory index is hot-path code
that a one-time pass over old rows does not justify.

Instead, when the pass changed anything, its summary log line tells the
operator to restart servers. The reasoning is recorded at the top of
`cmd/ingestor/hash_migrate.go`. (Raised by a reviewer in #2396.)

## Testing

- `cmd/ingestor/hash_migrate_test.go`: in-place rewrite; stale rows of
one scan batch commit in write-sized chunks
(`TestMigrateContentHashesCommitsInWriteChunks`); merge of a colliding
pair (observation move, dedup-index conflicts, span widening, child
cleanup); a stale pair split across batches; no-op; stops on cancel;
reports a write failure; scheduled on open; and
`TestContentHashMigrationSummary` (the restart hint, and silence when
nothing changed).
- `cmd/server/hash_migration_status_test.go`: `hashMigrationComplete`
follows the ingestor's migration row.
- `cmd/server/readonly_invariant_test.go`: no write path to the analyzer
DB from the server.
- Lock hold, measured on 100k stale transmissions (half merging, 5
observations each) with a concurrent writer probing the lock:

  | write batch | hold p50 | hold max | probe max wait | total |
  |---|---|---|---|---|
  | 5000 (before) | 1269 ms | 1356 ms | 1351 ms | 25.3 s |
  | 250 (now) | 63 ms | 108 ms | 158 ms | 27.5 s |

The same run at 10× the size (1M stale transmissions, 5M observations):

  | write batch | hold p50 | hold max | probe max wait | total |
  |---|---|---|---|---|
  | 5000 (before) | 1344 ms | 1808 ms | 1802 ms | 4 m 33 s |
  | 250 (now) | 66 ms | 132 ms | 199 ms | 4 m 38 s |
- Removed `cmd/server/hash_migrate_test.go` and
`hash_migrate_eviction_test.go` (#2327) along with the server code they
covered.
- `go vet` and `go test` pass on all modules, with gofmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`.

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected (zig already links them with 8 MiB thread stacks); it fixes
native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:16:29 +01:00

48 lines
1.6 KiB
Go

package main
import "testing"
// TestHashMigrationCompleteReadsIngestorStatus pins /api/stats
// hashMigrationComplete to the ingestor's _async_migrations row (#1856):
// false until the row says done, then latched true.
func TestHashMigrationCompleteReadsIngestorStatus(t *testing.T) {
db := setupTestDBv2(t)
if db.hashMigrationComplete() {
t.Fatal("missing _async_migrations table must read as not complete")
}
// The ingestor's bookkeeping table (cmd/ingestor/async_migration.go).
if _, err := db.conn.Exec(`CREATE TABLE _async_migrations (
name TEXT PRIMARY KEY, status TEXT NOT NULL, started_at TEXT, ended_at TEXT, error TEXT)`); err != nil {
t.Fatal(err)
}
if db.hashMigrationComplete() {
t.Fatal("missing row must read as not complete")
}
for _, status := range []string{"pending_async", "failed", "cancelled"} {
if _, err := db.conn.Exec(`INSERT OR REPLACE INTO _async_migrations (name, status) VALUES (?, ?)`,
contentHashMigrationName, status); err != nil {
t.Fatal(err)
}
if db.hashMigrationComplete() {
t.Fatalf("status %q must read as not complete", status)
}
}
if _, err := db.conn.Exec(`UPDATE _async_migrations SET status = 'done' WHERE name = ?`, contentHashMigrationName); err != nil {
t.Fatal(err)
}
if !db.hashMigrationComplete() {
t.Fatal("done must read as complete")
}
// done is final, so the answer is latched and the table is not read again.
if _, err := db.conn.Exec(`DROP TABLE _async_migrations`); err != nil {
t.Fatal(err)
}
if !db.hashMigrationComplete() {
t.Fatal("complete must stay latched once seen")
}
}