Undecodable stream values were acknowledged and deleted without recovery
evidence. They now go to a bounded quarantine stream (approx MAXLEN 500,
raw value size-capped, digest + byte length recorded) before the original is
acknowledged and deleted, so a malformed entry can neither block the stream
nor disappear without a trace. The structured log carries only id, length
and digest, and the quarantine counter is exported as
meshcore_ram_ingest_quarantined_total.
In RAM mode persistOnce returns at admission, so the post-ingest callbacks
that previously ran after durable persistence would fire against rows that
are not in PostgreSQL yet:
- the link queue (link worker reads packet rows) now enqueues from the RAM
writer after the batch commit, and the live packet subscriber skips it in
RAM mode;
- the live-map follow-ups (multibyte path evidence, slow-mode scheduling,
pre-resolution) move to the same post-commit hook, sharing the durable
path in-flight bound and drop counters via schedulePostPersistFollowUps;
- changes that are safe at admission stay there: WS broadcasts, node emits,
resolve-cache invalidation and owner forwarding (self-contained envelope).
Regression test: the hook fires with the persisted batch after bookkeeping,
and never on failed commits.
- Merged live reads now aggregate pending observations per packet hash, the
same way the SQL aggregates observers: one row per RF packet with combined
observer arrays, IATAs and RX/TX counts. A hash already present in the
database result is augmented with genuinely new observations only (an
observer already listed is treated as the same observation, so retained
overlap entries cannot double-count).
- The WS initial-state message history merge is restricted to group-text
packets (packet_type 5), matching the getRecentMessages contract, with
headroom in the pending sample so the type filter does not starve.
- Regression tests: multi-observer aggregation, augment-without-double-count,
and the type filter (messages vs all-types).
- Admission bounds (packet count, byte budget) are enforced in one atomic
Lua step with a conservative byte counter (value + fixed overhead);
concurrent MQTT connections can no longer race past the same capacity
check, refusals still never acknowledge, and the counter is kept honest
across crashes by release-at-commit.
- Oldest-pending gauge reads oldest-first: the previous XREVRANGE sample
reported the NEWEST entry age and could hide stuck deliveries.
- Committed entries are retained for a handoff-overlap window instead of
being deleted at commit, closing the gap where a reader sees neither a
pre-commit DB snapshot nor the post-commit stream. A retention sweep
removes entries only when provably safe: acknowledged (absent from the
PEL), at or below the newest commit boundary (numeric id comparison),
and past a minimum retention guard. Failed-batch PEL entries are never
swept.
- RAM writer commits idempotently from the first attempt: a crash after a
successful commit but before Redis bookkeeping replays the SAME frozen
entries on restart, so the identity existence check must apply immediately
rather than only on internal retries.
- Idempotent SQL gains an intra-batch identity dedupe (row_number over the
five-column identity key) so duplicate identities within one batch cannot
double-insert.
- All insert paths are gated behind a transaction-scoped advisory lock so
concurrent writers cannot both pass the NOT EXISTS check; contention is
surfaced as lock_ok=false and retried by the caller.
- Writer collapses duplicate identities before persistence; collapsed count
exported as meshcore_ram_ingest_batch_dedupe_total.
Admission-time visibility flags come from the prefix cache and can lag a node
privacy flip. Two hardenings: (1) the RAM writer refreshes the privacy cache
on a 30s cadence (piggybacked on its stats tick) so the window is bounded
even without traffic-driven refreshes; (2) merged public reads re-classify
each pending entry against the CURRENT cache and drop entries that are now
private (fails closed when the cache is unloaded). The database remains
authoritative for stored rows (fence + rematerialization); this removes the
pre-commit public-merge exposure window.
Startup PEL reclaim of a large leftover backlog could build a single flush far beyond the configured batch size; 26 bind parameters per row then exceeded the Postgres 65535-parameter limit (caught by the bench smoke as a one-off commit failure, recovered by the writer retry path). Flushes now take at most 2000 rows (52k parameters).
The RAM path adds observation_id to the incoming CTE; SELECT i.* then made the
bare observation_id reference ambiguous in downstream CTEs. The incoming column
is now observation_id_in (COALESCE(i.observation_id_in, gen_random_uuid())).
Caught by the bench smoke (real Postgres executes the batch SQL; unit mocks do
not). Validated via PREPARE against the bench database.
Adds MQTT_INGEST_MODE=ram: validated+classified packets are admitted to a
bounded Redis Stream and acknowledged at admission; an in-process writer
consumes the stream via a consumer group and persists batches through the
same SQL as the durable path (batch size or max-age flush trigger).
- admission is strict and fail-closed: queue bounds refuse rather than drop,
store unavailable = no acknowledgement (broker redelivers)
- commit handoff: XACK -> XDEL -> commit marker; observation_id is generated
at admission and carried through the batch INSERT for a stable identity
across the RAM -> PostgreSQL transition
- failed commits leave entries un-acked (PEL) and retry with backoff; writer
restarts reclaim the PEL (own 0 reads + XAUTOCLAIM)
- recent-read endpoints merge pending stream entries (public, deduplicated by
hash) so live views do not lag the flush window
- durable mode is the default and is behaviourally unchanged (same SQL via
persistEntriesCore; full suite green: 434/434)
- new metrics: queue depth/bytes/age, admissions, commit latency/visibility
latency, retries/reclaims
- Concurrent duplicates that arrive while the first persist is in flight now
re-arbitrate on failure: the successor claim is registered inside
waitForInFlightPacket in the same microtask as the emptiness check, so exactly
one waiter takes over and the rest queue behind it (previously two waiters
could both fall through after a failed attempt and persist concurrently,
risking a double insert since packets carry no unique constraint).
Regression tests cover the single- and chained-failure arbitrations.
- Post-persist follow-ups (multibyte evidence / slow-mode / pre-resolve) are now
bounded at 64 in flight; over the bound they are dropped and counted
(meshcore_mqtt_post_persist_dropped_total + _inflight gauge), so a database
stall cannot grow them without limit.
- MQTT_INGEST_CONNECTIONS (default 1) runs N durable ingest connections with a
shared subscription; each connection keeps strict PUBACK-after-commit
semantics - mqtt.js dispatches the next packet only after the previous
message is acknowledged, so a single connection is strictly serial and
concurrency has to come from connections.
- In-flight dedupe registry: a concurrent copy of a packet waits for the first
attempt outcome, preserving existing duplicate semantics under concurrency.
- QoS 1 redelivery guard: a redelivered message (duplicate flag) is confirmed
against the retained packet window before persisting, so a commit whose
PUBACK was lost to a crash is not double-counted after restart.
- Best-effort live-map follow-ups (multibyte evidence, slow resolution, cache
warm-up) move off the acknowledgement path.
- New metric: receipt-to-acknowledgement latency histogram.
- MQTT_INGEST_FLUSH_MS (default 10ms) replaces the fixed 50ms wait; the old
interval taxed every singleton batch, which is all a serial ingest produces.
- MQTT_INGEST_QUEUE_MAX (default 1000) bounds the pending queue; overflow
refuses the entry so the broker redelivers instead of growing memory.
- Duplicate packets inside one batch (same hash/observer/hop) resolve the
first entry outcome instead of being persisted twice - parity with the
in-memory ingest filter once more than one ingest can be in flight.
- New metrics: queue depth gauge, enqueue-to-commit wait histogram, dedupe counter.
* Fix analysis state, worker observability, feed layout and maintenance (#113)
* fix(analysis): restore tested retired workload policy
* fix(owner): prewarm last-hop cache with bounded progress reporting
* fix(metrics): use real link worker heartbeat for queue alerts
* fix(feed): measure virtual rows and preserve scroll anchors
* fix(ops): reject Compose metadata drift before replacement
* fix(maintenance): archive inactive nodes across roles and bridge states
* docs(review): record validation and heartbeat audit findings
* fix(maintenance): retain private identities during stale cleanup
* fix(ops): consume Compose service output before adoption checks
* test(owner): cover prewarm scheduling recovery and shutdown
* test(e2e): isolate server ports and Vite dependency caches
* test(feed): cover empty filters and arrivals at the top
* test(map): budget two cold bootstraps in mode persistence check
* docs(review): record continuation regressions and final verification
* fix(alerts): match queue heartbeats to their scrape reporter
* test(feed): preserve packet anchors across scroll-mode changes
* test(owner): verify incremental prewarm cache refreshes
* fix(maintenance): preserve packet privacy fence during public cleanup
* fix(maintenance): recheck inactive sightings before archive and delete
* fix(owner): serve fresh cache hits during background prewarm
* test(map): give each viewport its own startup budget
* test(map): load valid terrain and coverage tiles in local fixtures
* fix(metrics): export unknown heartbeat when Redis initialization fails
* test(path): control worker-pool deadlines independently of thread startup
* test(e2e): await owner and map readiness before state assertions
* test(feed): check fractional viewport and spacer invariants
* docs(review): record green expanded verification checkpoint
* fix(maintenance): acquire visibility before node locks and skip busy rows
* test(maintenance): retain concurrent RF-only activity in native fixtures
* fix(owner): replace regrouped last-hop buckets during refresh
* test(owner): verify warm regrouping with actual last-hop SQL
* test(maintenance): retain concurrently committed node consent
* test(maintenance): avoid a waiting consent-trigger deadlock
* test(feed): verify responsive anchoring in Firefox and WebKit
* docs(review): reconcile completed checks and remote CI evidence
* docs(review): correct checkpoint wording
* test(maintenance): verify privacy fences on native PostgreSQL
* test(health): verify link heartbeat publisher lifecycle
* docs(review): record stop-signal handoff and remaining verification
---------
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
* fix(feed): preserve scroll anchors across scroll-mode flips
React 19.3 fires the window scroll listener before React commits its new
row range, so updateViewport could overwrite the last known content offset
with the new-mode artifact (0) before the anchor-preserving layout effect
ran — dropping the feed to the top on a responsive resize.
Detect the scroll-mode flip (internal container <-> document scrolling) in
updateViewport and transfer the last known content offset into the new mode
before the anchor effect evaluates.
Verified: feed-virtualizer e2e (both viewports) and the full public-desktop
suite under react 19.2.8 and 19.3.0.
* fix(#553): sync replace-container test harness
* test(ops): sync replace-container harness with adoption + infra flows (#553)
Union of both branch evolutions: Compose adoption drills (check-compose-adoption
staging, MOCK_ADOPTION_* modes, adoption/long-service/overlay cases) plus the
infra-compose staging and timescaledb resolver drill assertions. Fixes the
Workers-and-Compose gate failing on this PR.
Verified locally: 16/16 drills pass.
---------
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
Co-authored-by: hermes-gadget <284726080+hermes-gadget@users.noreply.github.com>
* fix(mqtt): skip offline status telemetry samples
* docs: add MQTT Will telemetry handoff
* chore(db): renumber MQTT status migration to 057
052 is already taken on main by owner packet sharing; 055 predates main (unmerged branch) and is left untouched; 057 is the next free number after 056. Also appends the migration to the fresh-database expectation in packetBatch.integration.test.ts in sorted order.
---------
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
Co-authored-by: hermes-gadget <284726080+hermes-gadget@users.noreply.github.com>
* feat(ops): auto-rebaseline owner ACL inventory
* docs(ops): handoff owner baseline autorebaseline
* ops: add reviewed owner-authorization baseline refresh script
scripts/refresh-owner-baseline.sh regenerates the owner inventory baseline
with the running backend image, validates it against the live inventory
(fail-closed), backs up .env before repointing, and can recreate the backend.
It refuses to overwrite an existing baseline and keeps the previous file for
rollback. Adds the matching operations runbook section.
* test(ops): cover refresh baseline validation output
---------
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
The maintenance script ran bare `docker compose` in the repo dir, which
fails on the deployed layout where the database lives in the separate
meshcore-infra compose project ('db-migrate depends on undefined service
timescaledb'). Mirror scripts/backup.sh: when COMPOSE_FILE is unset,
target the meshcore-infra project (overridable via MESHCORE_INFRA_DIR /
MESHCORE_INFRA_PROJECT_NAME); the test harness keeps working because it
sets COMPOSE_FILE/COMPOSE_PROJECT_NAME (env path unchanged).
Verified: read-only chunk inventory runs on projects-vps; test-vacuum-
maintenance.sh fixture suite re-run.
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
gadgethd