`NativeHooks::sticky_backlog` loads a backlogged batch, cleaning up rooms
whose sticky events expired, reads the events through the shared event
cache and filters them for the destination. Its completion records the
batch natively. Where the filter would wait for full state, the batch is
left to Python, as before.
`FederationStore` also gains `get_catch_up_outstanding_destinations` for
phase 6. Each query has a differential test against Python's.
The native queue tests now run with MSC4354 on, so the loops look at the
sticky event backlog, and pin that it takes no GIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::catch_up` runs `_catch_up_transmission_loop` natively. It
reads the catch-up events and the room's forward extremities, loads
events through the shared event cache, filters the extremities with the
Rust `filter_events_for_server`, sends each room's PDUs through the
queue's sink, records the sticky event gap and moves the catch-up
position on. A destination catching up no longer takes the GIL, unless a
module wants to hear about delivered events.
`FederationStore` gains the catch-up queries, with differential tests
against Python's. The `catch_up` hook is given its queue, and a PDU can
now hold an event loaded in Rust.
Where Python would wait for a room's full state to filter, Rust sends
the original PDU, as it does for a room with partial state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Stop catch-up before the next room once the loop is shut down. The Rust
loop cannot cancel the Python catch-up coroutine, and after a shutdown
the transaction manager returns without sending, so the room would be
recorded as sent.
- Record the request metrics when the Python fallback for reading a
transaction response fails.
- Give each transmission loop its own LoopActivity, so a loop that ends
cannot clear the parked flag of the next.
- Count the time spent reading a hook's result as GIL time of its site.
- Correct the AwaitableObserver::finished doc.
- Let test_send_receipts_with_backoff_sender accept either order of its
first two sends, which run at once with the queues in Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`DestinationQueue` holds what waits to go to one destination: PDUs,
EDUs, keyed EDUs, presence and receipts, with Python's merge rules and
limits. The first thing queued starts its transmission loop as a tokio
task, which builds and sends transactions until there is nothing left,
as `_transaction_transmission_loop` does.
Catch-up, the sticky event backlog and the loop's database work stay
Python for now, behind `QueueHooks`. `RustDestinationQueue` is the
pyclass the Python facade wraps, and calls those hooks on it.
`run_python_awaitable_observed` converts a Python result on the reactor
and tells an observer when the awaitable starts and finishes. The loop
uses it so tests can tell a loop waiting on the reactor from one with
work to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
