- Re-raise SlidingSyncUnknownPosition from finish_request. A racing
request on the same connection can clear the position before the
persist, and Python would fail the same way, so a Python retry only
repeats the work.
- Treat an empty events stream cache as a miss, as Python's
get_entities_changed does, so Rust asks the database.
- Copy the room statuses out of the connection state before building
Python objects, so the lock is never held across a Python call.
- Trace the rooms Python computes for Rust, as the Python path does.
- Test the per-room fallback: Python computes a room Rust leaves to it,
and its changes reach the Rust state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
In `rust_sliding_sync: on` mode, `current_sync_for_user` is now four Rust
calls around the Python extensions: load the connection state, compute
the interested rooms (the port of `_compute_interested_rooms_new_tables`),
compute the rooms, and finish the request (unsent and sent rooms, then
the persist).
The connection state is a Rust pyclass. The extensions read and update it
through views with the surface of Python's `PerConnectionState` and
`MutablePerConnectionState`. Receipts tokens stay in their database form
in Rust and become `MultiWriterStreamToken`s only when Python asks.
The whole request falls back to Python when the sliding sync tables are
not filled, for a filter Python does not implement either, and on any
Rust error. Rooms Rust leaves to Python run `get_room_sync_data` over a
Python copy of their slice of the state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
The differential test now also checks that the Python and Rust sides
make the same changes to their connection's state: the room configs and
the lazy-loaded members. It also covers an avatar change, a ban, a
rejected remote invite, a state reset out of a room, and a prev_batch
token that names another writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
current_sync_for_user hands the rooms to send to Rust and applies the
connection-state changes it returns. Python computes the rooms Rust
leaves to it or fails on. The servlet encodes Rust rooms in Rust, and the
extensions read rooms through accessors both kinds of room have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
SlidingSyncHandler.generate_room_entries runs get_room_sync_data for a
batch of rooms, twenty at a time, and returns each room's entry with the
two changes it makes to the connection state. encode_rooms is the
servlet's encode_rooms for those entries.
A room's state loads reuse the events its timeline loaded, as Python's
held-event map does, so state events carry the same prev_content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
filter_events_for_client takes is_peeking, which v2 passes as false and
sliding sync sets for rooms the user is not joined to. Events gain
instance_name, and unsigned data gains invite_room_state and
knock_room_state accessors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`max_event_pos` now takes `get_max_pos_of_last_change` for the rooms the
events stream cache knows, as `_bulk_get_max_event_pos` does. The port
keeps Python's quirk: the cache answers only when it knows every missing
room, because Python's batch loop goes over all of them.
The membership change tests now add delta rows without an event, as the
server leaving a room or a state reset write them. The branches that
handle those rows were not reached before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`SlidingSyncReader` gains the reads and writes of a connection's
per-request state:
- `connection_lazy_members`: `get_sliding_sync_connection_lazy_members`
- `load_connection_state`: the `get_and_clear_connection_positions`
transaction. It bumps `last_used_ts`, deletes the other positions,
un-forks the lazy members, drops the unused required state rows and
reads the state.
- `persist_connection_state`: the `persist_per_connection_state`
transaction, from the row lock through the lazy member upserts, with
`UNNEST` for the batched upserts and deletes
The state crosses as `StoredConnectionState` and `ConnectionStateUpdates`,
with each stream's tokens as strings, as `PerConnectionStateDB` holds
them. Parsing the tokens is the caller's job once the state is Rust-owned.
An unknown position is the typed `UnknownPosition`, which the test entry
points raise as `SlidingSyncUnknownPosition`.
Both run as `REPEATABLE READ` read-write transactions, retried up to
five times on a serialization failure or a deadlock, as `runInteraction`
does. The caller passes the time in, so the tests drive both sides from
one clock. Loaded states are memoized by their arguments in a cache
named after Python's `@cached` one, which nothing invalidates. A
long-poll's second load and a client's retry therefore run the
transaction once.
Rooms share required state rows by their JSON. `encode_required_state`
writes it as Python's `json_encoder` does, ASCII-escaped, so that rows
written by either side dedup.
The test runs twelve random operations through Python on one connection
and through Rust on another, 20 seeds. The operations are persists on
the latest position, on a fork or from scratch, and loads of current,
stale and foreign positions. After each step it compares the two
connections' rows (positions by order, required state IDs by their
JSON) and what each load returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/storage/sliding_sync_store.rs` adds `SlidingSyncReader`. It is
a `SyncReader` with the sliding sync queries on top, so one request
counts its v2 and sliding sync reads in one set of stats. Only the path
Python takes once the background updates have finished is ported.
- membership snapshots: `rooms_from_snapshots`, `self_leave_rooms_after`,
`snapshot_room` and `snapshot_rooms`
- membership changes: `snapshot_membership_changes` and
`membership_delta_changes`. The caller passes the membership stream
cache's answer in as a bool.
- stream positions: `rooms_with_updates_since`, which takes the events
stream cache's changed rooms when it has them, and `max_event_pos` and
`last_event_pos_before`
- room metadata: `room_type`, `room_version_id`, `partial_rooms`,
`tags_for_user`, `cached_member_counts` and `is_encrypted_at`
Each Python `@cached` method behind these gets a Rust cache that mirrors
it by name: the snapshots, `_get_max_event_pos`, `get_member_counts`,
`get_room_type`, `get_room_version_id`, `get_partial_rooms`,
`get_tags_for_user` and `get_global_account_data_by_type_for_user`. The
v2 reader's global account data read now goes through that last one.
`get_partial_rooms` takes no arguments, so `()` becomes a cache key.
A row's null instance name reads as "master", as everywhere else in
Rust. Python keeps the `None` in `get_sliding_sync_membership_changes`.
That only matters for a token whose instance map names "master".
`tests/storage/test_sliding_sync_store.py` runs each query against the
Python method on the same database, with the writers scattered over
four instances for the token-bounded reads.
Python's `get_rooms_that_have_updates_since_sliding_sync_table` calls
`have_finished_sliding_sync_background_jobs()` without awaiting it. The
coroutine is always truthy, so Python never takes its "assume all rooms
changed" branch. The Rust port does not have that branch either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/extensions/relevant.rs` holds the matching of an
extension's `lists` and `rooms` against the response. Phase 5 uses it,
and it is cheap to port now. A list without exactly one `SYNC`
operation fails, as Python's assertion does: that is a list with no
ranges or several, under an extension that names it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room.rs` holds the parts of `get_room_sync_data`
that make no reads:
- the timeline bounds (`from_bound`, `initial`, the expanded timeline)
- `num_live`
- the stripped state of an invite or knock
- the string fields of the name, avatar and member events, and the
hero assembly
- `_get_bump_stamp` split around its two reads, plus the
`always_return_bump_stamp` rule and the clamp to zero
- the room config the connection records afterwards
Three of these move, unchanged, out of `get_room_sync_data` into Python
helpers, so the tests compare against the real code: `_timeline_bounds`,
`_stripped_state` and `_room_config_to_persist`. `_get_bump_stamp` is
tested as it is, with a stand-in store.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/sort.rs` holds `sort_rooms` as a
`SortPlan`: the decision logic with the stream change cache shortcut,
and the caller fetching what each step asks for. `lists.rs` holds the
ranges to `SYNC` operations with the full-range no-sort shortcut, the
config combining across lists and room subscriptions, and the decision
half of `_filter_relevant_rooms_to_send`, including the reset for a
connection with too many rooms to update.
The Python ranges loop moves, unchanged, into `_sync_list_ops`, so the
test can call it. It keeps a quirk the Rust ports: once a room combines
with a config from an earlier list, every later room in the same list
takes the combined config too.
The relevant room map is an `IndexMap`, which keeps Python's dict order,
so `indexmap` becomes a direct dependency. It was already in the tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/filter.rs` holds
`filter_membership_for_sync` and `filter_rooms_using_tables` over
`RoomForUser`, the Rust form of `RoomsForUserSlidingSync`. The caller
passes the DM rooms, the user's tags and whether the connection sent a
room before, which decides the state-reset keep rule. The `spaces` and
`room_name_like` filters return an error, as Python raises
`NotImplementedError` for them.
The filtered rooms keep their input order. Python's are in set order.
The differential test calls Python's method with a stand-in store over
2000 generated room sets and filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/connection.rs` holds `HaveSentRoom`,
`RoomStatusMap` and its mutable twin with the live-only transitions,
`PerConnectionState` and `MutablePerConnectionState`, and
`RoomLazyMembershipChanges` with the hourly last-seen rewrite. Python's
`ChainMap` becomes the previous maps behind an `Arc` plus the request's
updates, so persisting can write only the updates, as it does today.
The functions take the time as an argument. Phase 4 passes
`runtime.clock()`, which follows trial's virtual time. The receipts
stream's token type stays a parameter until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/required_state.rs` holds `_required_state_changes`
and the derivation in `get_room_sync_data` of the state filter and the
member sets from a room's `required_state` and timeline. The Python
derivation moves, unchanged, into two module-level helpers,
`_required_state_request` and `_state_filter_with_meta`, so the
differential tests can call it.
Python's results depend on set iteration order in three places, and Rust
takes one of Python's possible outcomes in each:
- The previous state keys kept over the 100-key cap: Python slices a
set, Rust takes the first in sorted order.
- Converting explicit members to lazy ones reads a loop variable that
leaked out of the loops before it. Without state deltas, Python's is
the last type of a set union in hash order, so whether those members
join the lazy cache varies between processes. Rust reads
`m.room.member`'s.
- `must_await_full_state` raises on an empty state key under the `*`
type, unless a remote user ID comes first in the set. Rust checks the
user IDs first.
The tests compare over 3000 generated inputs each, and allow exactly
these differences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/config.rs` holds `from_room_config`, `combine` and
`must_await_full_state`, the wildcard collapsing rules of a list's or a
room subscription's `required_state`. A differential test compares each
with the Python over 2000 generated configs.
`must_await_full_state` fails where Python raises: an empty state key
under the `*` type, which Python indexes into (`""[0]`).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Four reads on `SyncReader` for the per-room step of a later phase, each
checked against the Python store method in `tests/storage/test_sync_store.py`:
- `state_deltas_by_stream_id`: `get_current_state_deltas_for_room`, bounded
on the delta rows' own `stream_id`. Sliding sync reads this variant, not
the by-event-position one the v2 port uses.
- `last_event_of_types_before`: `get_last_event_pos_in_room_before_stream_ordering`
with `event_types`, for the bump stamp. `last_event_before` calls it with
no types.
- `member_counts`: `get_member_counts`.
- `latest_bump_stamp`: `get_latest_bump_stamp_for_room`.
None is cached yet. The plan mirrors `get_member_counts` by name in phase 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Python validates the body with pydantic as before. Rust parses the same
bytes with serde into `SlidingSyncBody`, which follows the pydantic model
field for field, including where it tells an explicit `null` from a missing
field. Lists and room subscriptions keep document order, and a repeated key
keeps its first position and last value, as the dict `json.loads` builds.
`SlidingSyncRequest` is the body plus what Python adds: the user and device,
the room key and connection position of `pos`, the room key and string suffix
of the token the response runs up to, the admin's soft-failed policy, whether
the background updates have finished, the excluded rooms and the extension
flags. `SlidingSyncHandler._build_rust_request` builds it.
The servlet keeps the body's bytes in `SlidingSyncConfig.body_bytes` when the
Rust handler is in use. A config built without a request is re-encoded.
`tests/rest/client/sliding_sync/test_rust_request.py` parses 400 generated
bodies (unknown fields, nulls, repeated keys, escapes) both ways and
compares, and checks the re-encoding parses the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Phase 0 of the sliding sync port (renders/sliding-sync-rust-plan.md). No
behaviour change: nothing calls the Rust handler yet.
- `experimental_features.rust_sliding_sync: off | on`. There is no shadow
mode. Booleans are accepted, as for `rust_sync`.
- `synapse_rust.sliding_sync.SlidingSyncHandler`, built by `RustHandlers`
on the native driver only. It shares the v2 sync handler's `SyncStore`
and carries the pool, deferred and blocking helpers the later phases'
entry points use.
- `SlidingSyncStats`, observed per request into the
`synapse_sliding_sync_rust_*` histograms. `SyncDbStats` now counts the
transactions a reader begins.
- The `synapse_sliding_sync_rust_rooms_total` and
`synapse_sliding_sync_rust_requests_total` counters.
- `SYNAPSE_TEST_RUST_SLIDING_SYNC` and `_STRICT` for trial, with the handler
wrapped in `_BlockingRustCalls`, and `RUST_SLIDING_SYNC` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
With the main database on the native Postgres driver, every event load
goes through the Rust `EventLoader`. `EventsWorkerStore` sends the events
its own lookups miss to `RustStore.load_events`, and `_get_events_from_db`
to `reload_events`. Loads from Python, sync and state resolution then
coalesce in the shared event cache, and an invalidation during a load
stops it caching a stale entry. Python's fetch threads and in-flight map
stay only for other databases.
- `EventLoader::load` goes through `Cache::get_or_load_many`. The extra
redaction events it reads are returned but no longer cached.
- `PoolQuerier` checks its connection out before claiming any loads, so
readers that hold connections while they wait cannot starve a load.
- An unusable event raises `InvalidEventError` or
`DatabaseCorruptionError`, as from Python, which some callers catch.
- Cancelling a Python caller leaves the load running for the others, and
`evt_db_fetch_count` counts only what a caller read itself.
- `RustStore` holds the `DatabasePool` weakly and reads its pool per call.
Tests: trial makes the loads block, as it does Python's transactions
(`_BlockingRustCalls`, renamed from `_BlockingRustSyncHandler`). The
cancellation and outage tests get Rust variants. Two tests stop depending
on how far requests advance the clock or on which `DatabasePool` object
a worker store keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
`Cache::get_or_load_many` looks up several keys at once and loads the
misses with one call. Each key coalesces with loads already in flight,
single or batched, and a key invalidated mid-load is answered but not
cached. Waiters now get `None` for a key a batch load did not find, and
a single load waiting on such a key loads it itself.
`Cache::abandon_loads` stops every in-flight load from caching, for an
invalidation that cannot name the keys it affects. A room purge uses it
on the event cache, as the room of an event being loaded isn't known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
- The server filter read each restricted event's state group before it
checked for partial state, and through a cache that a prefill never
reached. After an event got its full state, a retry could use the old
partial group. It now checks first, then reads those groups from the
database. A prefill of a Python cache now invalidates its Rust mirrors.
- Catch-up ended in two steps, so a PDU that send_pdu skipped in between
was lost until the next catch-up. One lock now covers both.
- The sticky backlog query reads a fresh snapshot per statement, so
events could expire between its cleanup and its retry. It now cleans
up again, a few times, before giving up.
- Tests for each, plus catch-up's batch limit, a failed catch-up send,
module notification, a filtered-out sticky batch, the hand-off to
Python, the backlog cleanup boundary and paging outstanding
destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::sticky_backlog` loads a backlogged batch, cleaning up rooms
whose sticky events expired, reads the events through the shared event
cache and filters them for the destination. Its completion records the
batch natively. Where the filter would wait for full state, the batch is
left to Python, as before.
`FederationStore` also gains `get_catch_up_outstanding_destinations` for
phase 6. Each query has a differential test against Python's.
The native queue tests now run with MSC4354 on, so the loops look at the
sticky event backlog, and pin that it takes no GIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::catch_up` runs `_catch_up_transmission_loop` natively. It
reads the catch-up events and the room's forward extremities, loads
events through the shared event cache, filters the extremities with the
Rust `filter_events_for_server`, sends each room's PDUs through the
queue's sink, records the sticky event gap and moves the catch-up
position on. A destination catching up no longer takes the GIL, unless a
module wants to hear about delivered events.
`FederationStore` gains the catch-up queries, with differential tests
against Python's. The `catch_up` hook is given its queue, and a PDU can
now hold an event loaded in Rust.
Where Python would wait for a room's full state to filter, Rust sends
the original PDU, as it does for a room with partial state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`visibility::filter_events_for_server` reads the erased senders, the
partial state rooms, the history visibility and the target server's
memberships at each event through a `SyncReader`, and leaves the
per-event decision to `event_visible_to_server`. That now borrows its
arguments.
Python waits for the full state of an event whose memberships it needs.
Rust cannot, and returns `NeedsFullState` for the caller to handle.
The differential tests compare it with Python's on random rooms, an
outlier and a room with partial state, for every combination of flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Keep an unchanged device stream's position. The tokens are read after
the change check, so moving to them could skip a message that arrived
in between.
- Decode cross-signing keys as leniently as Python does, including
URL-safe base64.
- Mark the transaction open before awaiting BEGIN, so a dropped task
still rolls back.
- Tell modules about delivered events before the database writes, as
Python does.
- Update the stream change cache before its listeners, and log a
listener that fails.
- Check the metrics through a registry scrape, with a fresh event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`FederationStore` ports the store methods the federation transmission
loop calls around each transaction to the native Postgres driver:
`get_new_device_msgs_for_remote`, `delete_device_msgs_for_remote`,
`get_device_updates_by_remote` (with the cross-signing key updates and
cross-signatures), `mark_as_sent_devices_by_remote` and
`set_destination_last_successful_stream_ordering`. The stream change
cache checks those start with are left to the caller.
None of the writes has a Python cache to invalidate. The cross-signing
reads skip Python's caches.
`tests/storage/test_federation_store.py` runs each against the Python
method on one database, through a `FederationStoreForTest` pyclass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Stop catch-up before the next room once the loop is shut down. The Rust
loop cannot cancel the Python catch-up coroutine, and after a shutdown
the transaction manager returns without sending, so the room would be
recorded as sent.
- Record the request metrics when the Python fallback for reading a
transaction response fails.
- Give each transmission loop its own LoopActivity, so a loop that ends
cannot clear the parked flag of the next.
- Count the time spent reading a hook's result as GIL time of its site.
- Correct the AwaitableObserver::finished doc.
- Let test_send_receipts_with_backoff_sender accept either order of its
first two sends, which run at once with the queues in Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`FederationSender` makes a `RustPerDestinationQueue` for each
destination at the new `queues` level. It forwards the queue methods to
Rust and provides the hooks: catch-up, the sticky backlog, the device
EDUs, and recording a sent transaction (stream positions, the module
callback, the metrics).
The catch-up loop moves into `_CatchUpLoop`, which both queues share.
`StickyEventBacklogTracker.load_backlog` splits the database part out of
`prepare_transaction`.
The sender, catch-up and sticky event tests also run at this level, with
every reactor advance waiting for the Rust loops to settle.
`test_send_receipts_thread` is skipped there: it depends on coroutine
order, which the Rust loop changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`DestinationQueue` holds what waits to go to one destination: PDUs,
EDUs, keyed EDUs, presence and receipts, with Python's merge rules and
limits. The first thing queued starts its transmission loop as a tokio
task, which builds and sends transactions until there is nothing left,
as `_transaction_transmission_loop` does.
Catch-up, the sticky event backlog and the loop's database work stay
Python for now, behind `QueueHooks`. `RustDestinationQueue` is the
pyclass the Python facade wraps, and calls those hooks on it.
`run_python_awaitable_observed` converts a Python result on the reactor
and tells an observer when the awaitable starts and finishes. The loop
uses it so tests can tell a loop waiting on the reactor from one with
work to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Only build the Rust sender from a real `RustFederationClient`. A test
that mocks the federation client no longer fails with a `TypeError`.
- Log the `TX` and response lines under Python's logger names, so log
configs that name those loggers still apply.
- Move the PDUs and EDUs into the request body, so a transaction holds
one copy of them while it is retried, besides the canonical JSON.
- Drop the unused `time_now` from `Event::to_pdu_value`.
- Set the compare mode per `TransactionManager` in the tests that use
it, not on the class.
- Test that the tracing headers reach the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
At the new `transactions` level, `TransactionManager` hands each
transaction to the Rust sender. Python keeps the tracing span, the
module callback and the `last_sent_pdu_time` gauge.
In tests, every transaction body Rust builds is checked against the one
Python builds (`SYNAPSE_TEST_RUST_FED_STRICT`). The federation sender
and catch-up test cases also run at the new level, with Rust building
the transactions that their mocked `send_transaction` receives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`TransactionSender` builds `/send` transactions and sends them through
the Rust federation client, with the options `send_transaction` uses.
It reads the response and returns the ids of the PDUs the remote
accepted. `Event::to_pdu_value` builds the PDU JSON from the Rust event
without the GIL.
`RustTransactionManager` is the Python facade. It takes the events and
EDU dicts, and returns a Deferred. A response body that serde_json
refuses goes to Python's decoder, as `JsonParser` would.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Create the sleeper's `Notified` under the map lock, so a wake that lands
just before the sleep starts is not lost. Count sleeps per destination
and forget the destination in a drop guard, which also runs on cancel.
- Map an SRV `.` target to `ConnectError`, as Python's resolver does.
- Call a custom parser's `finish` when its last write fails, and when the
request is cancelled mid-read.
- Check the path is ASCII after the backoff check, as Python does.
- `put_json` takes `rewrite_age_ts`, which `send_transaction` sets. Any
other `json_data_callback` sends through the Python client.
- Count attempts in the no-retry test, and test the request counters,
the proxy-worker fallback, shutdown and the server-up wake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Add the `experimental_features.rust_federation_sending` level, `off` or
`client` for now. At `client`, `MatrixFederationHttpClient` sends its
JSON requests (`put_json`, `post_json`, `get_json`,
`get_json_with_headers`, `delete_json`, `send_direct_request`) through
the Rust client. Media downloads, and an instance that sends through a
proxy worker, stay on the Python path.
The facade keeps the tracing span around the call and injects its
headers. The request and response counters are bumped from the GIL hop
that returns the result. The limiter's background write is factored out
of `RetryDestinationLimiter` so both paths share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Port `MatrixFederationHttpClient._send_request` and `_handle_response`
to Rust, over the phase 1 agent: destination validation and the
whitelist, the retry limiter, `X-Matrix` signing per attempt, the long
and short retry schedules with jitter, `wake_destination`, timeouts on
the virtual clock, the trailing slash retry, and the mapping of each
failure to the exception Python raises.
The JSON parsers run on tokio without the GIL. Python's decoder is the
fallback for the JSON serde_json refuses, such as lone surrogates. A
custom `ByteParser` is fed under the GIL in 64 KiB chunks. The
transaction manager's `json_data_callback` becomes a `rewrite_age_ts`
option, so a retry needs no call into Python.
The retry timings are read through a Rust mirror of
`get_destination_retry_timings`. Writes go to a Python callable, which
invalidates the cache across workers and reports a server coming back.
Every GIL acquisition in the federation code goes through `with_gil`,
which counts calls and hold time per call site.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
A shared .well-known lookup only ran while a caller awaited it. If every
caller gave up, the next request resumed it with its fetch timeout long
expired, and cached a failure for an hour. Run it as its own task.
The TLS handshake and a proxy's CONNECT answer had no timeout. hyper
finishes some connections in a background task that no request timeout
reaches, so a peer that stalled after accepting leaked the task and its
socket. Give each 30 seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The Rust replacement for `MatrixFederationAgent` and its connection pool,
as a library: nothing in Synapse sends through it yet. The request layer
(signing, retries, backoff) comes next and will be its first caller.
`Agent::request` takes a `matrix-federation://` request, applies
`.well-known` delegation to the URI and the `Host` header, and sends it
through hyper-util's pooled client (5 idle connections per host, 120s
idle timeout, HTTP/1.1, no automatic retries), as Python's pool does.
The pool calls our connector, derived from matrix-hyper-federation-
client's `MatrixConnector` (Apache-2.0). It resolves the server name
(SRV), tries each server in order and returns the first error if all
fail. For each server it resolves addresses through the IP range policy,
connects TCP with Twisted's staggered attempts and 30s timeout, or
tunnels through `https_proxy` with `CONNECT` unless `no_proxy` exempts
the host, then does TLS verified against the delegated server name,
never the SRV target. The `.well-known` fetch shares the pool and, as in
Python, blocks IP literals by the blocklist alone.
Failures are a typed `TransportError` (DNS, connect, TLS, and so on), so
the request layer can tell a DNS failure from the rest. The agent
implements `FederationTransport`.
`RustFederationAgentForTest` exposes a blocking `request` to Python. The
trial tests drive it against a real TLS server on a thread, signed by
the test CA, covering the delegation, SRV, SNI, IP literal, blocklist,
certificate verification and connection reuse cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The parts of the outbound federation agent that decide where to connect
and what to trust, each a port of the Python it replaces, with Rust unit
tests. Nothing uses them yet.
- `resolve`: `_matrix-fed._tcp`, then `_matrix._tcp`, then port 8448, as
`MatrixHostnameEndpoint._resolve_server`. SRV records are ordered as
`_sort_server_list` (RFC 2782), cached while every record is live, and
the stale answer is used when a refetch fails. A lone `.` target means
the service is unavailable. SRV lookups go through hickory, and A/AAAA
through the system resolver, as Twisted uses getaddrinfo. Parts derive
from matrix-hyper-federation-client (Apache-2.0).
- `well_known`: `WellKnownResolver`, with Python's cache periods, jitter,
grace period, had-valid memory, size cap and retries, behind a
`WellKnownFetcher` trait so the state machine is tested without HTTP.
Two additions: each fetch attempt has a timeout (`client_timeout`), and
concurrent lookups for one server name share a fetch. An `m.server`
that is not a valid URI authority now counts as an invalid response.
- `policy`: the IP range lists, refusing a name if any of its addresses
is blocked (the DNS rebinding defence), and the `https_proxy`
settings with Python's `no_proxy` matching.
- `tls`: `FederationPolicyForHTTPS` on rustls: platform roots or the
custom CA list, verification off (still sending SNI), the per-host
verification whitelist. rustls has no TLS 1.0 or 1.1, so a minimum
below 1.2 means 1.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
A dropped Clock::sleep left its waker in VIRTUAL_SLEEPERS until the
virtual time passed its deadline, which a later test case's reactor may
never do. Prune closed wakers when sleeping and when the time moves.
Nothing constructs IpBlocked: a blocked address is a failed DNS lookup,
as in Python.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Extend the Rust view of the homeserver config with what the federation
request layer and agent need:
- `config.federation`: timeouts, retry counts and delays, destination
backoff, the domain whitelist and the metrics domains.
- `config.tls`: certificate verification, the per-host verification
whitelist, the minimum TLS version, and the custom CA list. Python now
keeps the PEM text of that list too, so Rust never reads the files.
- `config.server`: the federation IP range block and allow lists (taken
from `netaddr.IPSet`s), the user agent suffix and the proxy config.
The verification whitelist holds Python regexes made by
`glob_to_regex`. They translate to Rust's syntax with two changes: `\Z`
becomes `\z` and `re.IGNORECASE` becomes `(?i)`.
The changelog entry needs renaming to the PR number.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Add `rust/src/federation`, starting with the `X-Matrix` request
signature and the transport seam the request layer will send through.
The signing code comes from matrix-hyper-federation-client (Apache-2.0),
changed to use Synapse's canonical JSON serializer, Python's header
layout and quoting, and `destination_is` for identity servers. It keeps
that crate's signature test vectors. The body is serialized to canonical
JSON once, then both signed and sent in that form.
The homeserver's signing key crosses to Rust as its algorithm, version
and seed, taken from `hs.signing_key`. Rust never reads the key file.
`FederationTransport` is what the request layer will send through.
`PyTransport` implements it with a Python callable, which is how Rust
level tests will inject responses in place of `FakeTransport`.
A trial test checks the Rust header is byte-identical to the one
`MatrixFederationHttpClient.build_auth_headers` builds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Tokio's timers run on real time, but trial drives a virtual reactor clock
that only moves when a test advances it. A Rust task that sleeps with
`tokio::time::sleep` under trial waits out real time while the test
believes none has passed.
`Clock::sleep` sleeps on the virtual clock while a test has pinned it:
`set_virtual_time_msec` wakes the sleepers that are due. Otherwise it is
`tokio::time::sleep`. Code that sleeps takes a `Sleeper`, so that its
Rust unit tests can use tokio's pausable clock instead, since the
virtual time is process-global and an earlier test may have pinned it.
The federation sender port needs this for its backoff and well-known
retry sleeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The new `sync/relations.rs` ports the relations handler's
`get_bundled_aggregations`: thread summaries with participation and
ignored users' replies subtracted, references, and the latest edit. The
Rust sync handler bundles them for limited and newly joined timelines,
loads the redactions of the bundled events, and the encoder serializes
them. Python no longer bundles the rooms Rust computes, and the room
entries hand Python the bundles only when something reads the timeline.
A thread whose latest reply is from an ignored user falls back to
Python for that room: finding the latest reply that is not takes the
relations pagination, which is not ported.
The relations queries read the database each time. Python's relations
caches are not mirrored.
Shadow mode now compares the Rust bundles with Python's, in the encoded
rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The rooms the Rust sync handler computes now reach the servlet as
`RustJoinedSyncResult` and `RustArchivedSyncResult`. They subclass the
Python entries, and their fields become Python objects only when read.
The servlet encodes them in one call to the new Rust `RoomsEncoder`,
which ports `encode_room` over the existing serializer. If that call
fails, Python encodes them.
The Rust handler loads each room's redactions as it computes the room.
The serializer reads redactions, bundles and module additions through a
`SerializeLookups` trait, so that it takes Rust events as well as Python
maps. Bundled aggregations are still computed in Python.
Shadow mode now also compares the encoded JSON of each room.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
