In `rust_sliding_sync: on` mode, `current_sync_for_user` is now four Rust
calls around the Python extensions: load the connection state, compute
the interested rooms (the port of `_compute_interested_rooms_new_tables`),
compute the rooms, and finish the request (unsent and sent rooms, then
the persist).
The connection state is a Rust pyclass. The extensions read and update it
through views with the surface of Python's `PerConnectionState` and
`MutablePerConnectionState`. Receipts tokens stay in their database form
in Rust and become `MultiWriterStreamToken`s only when Python asks.
The whole request falls back to Python when the sliding sync tables are
not filled, for a filter Python does not implement either, and on any
Rust error. Rooms Rust leaves to Python run `get_room_sync_data` over a
Python copy of their slice of the state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
The differential test now also checks that the Python and Rust sides
make the same changes to their connection's state: the room configs and
the lazy-loaded members. It also covers an avatar change, a ban, a
rejected remote invite, a state reset out of a room, and a prev_batch
token that names another writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`max_event_pos` now takes `get_max_pos_of_last_change` for the rooms the
events stream cache knows, as `_bulk_get_max_event_pos` does. The port
keeps Python's quirk: the cache answers only when it knows every missing
room, because Python's batch loop goes over all of them.
The membership change tests now add delta rows without an event, as the
server leaving a room or a state reset write them. The branches that
handle those rows were not reached before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`SlidingSyncReader` gains the reads and writes of a connection's
per-request state:
- `connection_lazy_members`: `get_sliding_sync_connection_lazy_members`
- `load_connection_state`: the `get_and_clear_connection_positions`
transaction. It bumps `last_used_ts`, deletes the other positions,
un-forks the lazy members, drops the unused required state rows and
reads the state.
- `persist_connection_state`: the `persist_per_connection_state`
transaction, from the row lock through the lazy member upserts, with
`UNNEST` for the batched upserts and deletes
The state crosses as `StoredConnectionState` and `ConnectionStateUpdates`,
with each stream's tokens as strings, as `PerConnectionStateDB` holds
them. Parsing the tokens is the caller's job once the state is Rust-owned.
An unknown position is the typed `UnknownPosition`, which the test entry
points raise as `SlidingSyncUnknownPosition`.
Both run as `REPEATABLE READ` read-write transactions, retried up to
five times on a serialization failure or a deadlock, as `runInteraction`
does. The caller passes the time in, so the tests drive both sides from
one clock. Loaded states are memoized by their arguments in a cache
named after Python's `@cached` one, which nothing invalidates. A
long-poll's second load and a client's retry therefore run the
transaction once.
Rooms share required state rows by their JSON. `encode_required_state`
writes it as Python's `json_encoder` does, ASCII-escaped, so that rows
written by either side dedup.
The test runs twelve random operations through Python on one connection
and through Rust on another, 20 seeds. The operations are persists on
the latest position, on a fork or from scratch, and loads of current,
stale and foreign positions. After each step it compares the two
connections' rows (positions by order, required state IDs by their
JSON) and what each load returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/storage/sliding_sync_store.rs` adds `SlidingSyncReader`. It is
a `SyncReader` with the sliding sync queries on top, so one request
counts its v2 and sliding sync reads in one set of stats. Only the path
Python takes once the background updates have finished is ported.
- membership snapshots: `rooms_from_snapshots`, `self_leave_rooms_after`,
`snapshot_room` and `snapshot_rooms`
- membership changes: `snapshot_membership_changes` and
`membership_delta_changes`. The caller passes the membership stream
cache's answer in as a bool.
- stream positions: `rooms_with_updates_since`, which takes the events
stream cache's changed rooms when it has them, and `max_event_pos` and
`last_event_pos_before`
- room metadata: `room_type`, `room_version_id`, `partial_rooms`,
`tags_for_user`, `cached_member_counts` and `is_encrypted_at`
Each Python `@cached` method behind these gets a Rust cache that mirrors
it by name: the snapshots, `_get_max_event_pos`, `get_member_counts`,
`get_room_type`, `get_room_version_id`, `get_partial_rooms`,
`get_tags_for_user` and `get_global_account_data_by_type_for_user`. The
v2 reader's global account data read now goes through that last one.
`get_partial_rooms` takes no arguments, so `()` becomes a cache key.
A row's null instance name reads as "master", as everywhere else in
Rust. Python keeps the `None` in `get_sliding_sync_membership_changes`.
That only matters for a token whose instance map names "master".
`tests/storage/test_sliding_sync_store.py` runs each query against the
Python method on the same database, with the writers scattered over
four instances for the token-bounded reads.
Python's `get_rooms_that_have_updates_since_sliding_sync_table` calls
`have_finished_sliding_sync_background_jobs()` without awaiting it. The
coroutine is always truthy, so Python never takes its "assume all rooms
changed" branch. The Rust port does not have that branch either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/extensions/relevant.rs` holds the matching of an
extension's `lists` and `rooms` against the response. Phase 5 uses it,
and it is cheap to port now. A list without exactly one `SYNC`
operation fails, as Python's assertion does: that is a list with no
ranges or several, under an extension that names it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room.rs` holds the parts of `get_room_sync_data`
that make no reads:
- the timeline bounds (`from_bound`, `initial`, the expanded timeline)
- `num_live`
- the stripped state of an invite or knock
- the string fields of the name, avatar and member events, and the
hero assembly
- `_get_bump_stamp` split around its two reads, plus the
`always_return_bump_stamp` rule and the clamp to zero
- the room config the connection records afterwards
Three of these move, unchanged, out of `get_room_sync_data` into Python
helpers, so the tests compare against the real code: `_timeline_bounds`,
`_stripped_state` and `_room_config_to_persist`. `_get_bump_stamp` is
tested as it is, with a stand-in store.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/sort.rs` holds `sort_rooms` as a
`SortPlan`: the decision logic with the stream change cache shortcut,
and the caller fetching what each step asks for. `lists.rs` holds the
ranges to `SYNC` operations with the full-range no-sort shortcut, the
config combining across lists and room subscriptions, and the decision
half of `_filter_relevant_rooms_to_send`, including the reset for a
connection with too many rooms to update.
The Python ranges loop moves, unchanged, into `_sync_list_ops`, so the
test can call it. It keeps a quirk the Rust ports: once a room combines
with a config from an earlier list, every later room in the same list
takes the combined config too.
The relevant room map is an `IndexMap`, which keeps Python's dict order,
so `indexmap` becomes a direct dependency. It was already in the tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/filter.rs` holds
`filter_membership_for_sync` and `filter_rooms_using_tables` over
`RoomForUser`, the Rust form of `RoomsForUserSlidingSync`. The caller
passes the DM rooms, the user's tags and whether the connection sent a
room before, which decides the state-reset keep rule. The `spaces` and
`room_name_like` filters return an error, as Python raises
`NotImplementedError` for them.
The filtered rooms keep their input order. Python's are in set order.
The differential test calls Python's method with a stand-in store over
2000 generated room sets and filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/connection.rs` holds `HaveSentRoom`,
`RoomStatusMap` and its mutable twin with the live-only transitions,
`PerConnectionState` and `MutablePerConnectionState`, and
`RoomLazyMembershipChanges` with the hourly last-seen rewrite. Python's
`ChainMap` becomes the previous maps behind an `Arc` plus the request's
updates, so persisting can write only the updates, as it does today.
The functions take the time as an argument. Phase 4 passes
`runtime.clock()`, which follows trial's virtual time. The receipts
stream's token type stays a parameter until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/required_state.rs` holds `_required_state_changes`
and the derivation in `get_room_sync_data` of the state filter and the
member sets from a room's `required_state` and timeline. The Python
derivation moves, unchanged, into two module-level helpers,
`_required_state_request` and `_state_filter_with_meta`, so the
differential tests can call it.
Python's results depend on set iteration order in three places, and Rust
takes one of Python's possible outcomes in each:
- The previous state keys kept over the 100-key cap: Python slices a
set, Rust takes the first in sorted order.
- Converting explicit members to lazy ones reads a loop variable that
leaked out of the loops before it. Without state deltas, Python's is
the last type of a set union in hash order, so whether those members
join the lazy cache varies between processes. Rust reads
`m.room.member`'s.
- `must_await_full_state` raises on an empty state key under the `*`
type, unless a remote user ID comes first in the set. Rust checks the
user IDs first.
The tests compare over 3000 generated inputs each, and allow exactly
these differences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/config.rs` holds `from_room_config`, `combine` and
`must_await_full_state`, the wildcard collapsing rules of a list's or a
room subscription's `required_state`. A differential test compares each
with the Python over 2000 generated configs.
`must_await_full_state` fails where Python raises: an empty state key
under the `*` type, which Python indexes into (`""[0]`).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Four reads on `SyncReader` for the per-room step of a later phase, each
checked against the Python store method in `tests/storage/test_sync_store.py`:
- `state_deltas_by_stream_id`: `get_current_state_deltas_for_room`, bounded
on the delta rows' own `stream_id`. Sliding sync reads this variant, not
the by-event-position one the v2 port uses.
- `last_event_of_types_before`: `get_last_event_pos_in_room_before_stream_ordering`
with `event_types`, for the bump stamp. `last_event_before` calls it with
no types.
- `member_counts`: `get_member_counts`.
- `latest_bump_stamp`: `get_latest_bump_stamp_for_room`.
None is cached yet. The plan mirrors `get_member_counts` by name in phase 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Python validates the body with pydantic as before. Rust parses the same
bytes with serde into `SlidingSyncBody`, which follows the pydantic model
field for field, including where it tells an explicit `null` from a missing
field. Lists and room subscriptions keep document order, and a repeated key
keeps its first position and last value, as the dict `json.loads` builds.
`SlidingSyncRequest` is the body plus what Python adds: the user and device,
the room key and connection position of `pos`, the room key and string suffix
of the token the response runs up to, the admin's soft-failed policy, whether
the background updates have finished, the excluded rooms and the extension
flags. `SlidingSyncHandler._build_rust_request` builds it.
The servlet keeps the body's bytes in `SlidingSyncConfig.body_bytes` when the
Rust handler is in use. A config built without a request is re-encoded.
`tests/rest/client/sliding_sync/test_rust_request.py` parses 400 generated
bodies (unknown fields, nulls, repeated keys, escapes) both ways and
compares, and checks the re-encoding parses the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Phase 0 of the sliding sync port (renders/sliding-sync-rust-plan.md). No
behaviour change: nothing calls the Rust handler yet.
- `experimental_features.rust_sliding_sync: off | on`. There is no shadow
mode. Booleans are accepted, as for `rust_sync`.
- `synapse_rust.sliding_sync.SlidingSyncHandler`, built by `RustHandlers`
on the native driver only. It shares the v2 sync handler's `SyncStore`
and carries the pool, deferred and blocking helpers the later phases'
entry points use.
- `SlidingSyncStats`, observed per request into the
`synapse_sliding_sync_rust_*` histograms. `SyncDbStats` now counts the
transactions a reader begins.
- The `synapse_sliding_sync_rust_rooms_total` and
`synapse_sliding_sync_rust_requests_total` counters.
- `SYNAPSE_TEST_RUST_SLIDING_SYNC` and `_STRICT` for trial, with the handler
wrapped in `_BlockingRustCalls`, and `RUST_SLIDING_SYNC` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
With the main database on the native Postgres driver, every event load
goes through the Rust `EventLoader`. `EventsWorkerStore` sends the events
its own lookups miss to `RustStore.load_events`, and `_get_events_from_db`
to `reload_events`. Loads from Python, sync and state resolution then
coalesce in the shared event cache, and an invalidation during a load
stops it caching a stale entry. Python's fetch threads and in-flight map
stay only for other databases.
- `EventLoader::load` goes through `Cache::get_or_load_many`. The extra
redaction events it reads are returned but no longer cached.
- `PoolQuerier` checks its connection out before claiming any loads, so
readers that hold connections while they wait cannot starve a load.
- An unusable event raises `InvalidEventError` or
`DatabaseCorruptionError`, as from Python, which some callers catch.
- Cancelling a Python caller leaves the load running for the others, and
`evt_db_fetch_count` counts only what a caller read itself.
- `RustStore` holds the `DatabasePool` weakly and reads its pool per call.
Tests: trial makes the loads block, as it does Python's transactions
(`_BlockingRustCalls`, renamed from `_BlockingRustSyncHandler`). The
cancellation and outage tests get Rust variants. Two tests stop depending
on how far requests advance the clock or on which `DatabasePool` object
a worker store keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
- The server filter read each restricted event's state group before it
checked for partial state, and through a cache that a prefill never
reached. After an event got its full state, a retry could use the old
partial group. It now checks first, then reads those groups from the
database. A prefill of a Python cache now invalidates its Rust mirrors.
- Catch-up ended in two steps, so a PDU that send_pdu skipped in between
was lost until the next catch-up. One lock now covers both.
- The sticky backlog query reads a fresh snapshot per statement, so
events could expire between its cleanup and its retry. It now cleans
up again, a few times, before giving up.
- Tests for each, plus catch-up's batch limit, a failed catch-up send,
module notification, a filtered-out sticky batch, the hand-off to
Python, the backlog cleanup boundary and paging outstanding
destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::sticky_backlog` loads a backlogged batch, cleaning up rooms
whose sticky events expired, reads the events through the shared event
cache and filters them for the destination. Its completion records the
batch natively. Where the filter would wait for full state, the batch is
left to Python, as before.
`FederationStore` also gains `get_catch_up_outstanding_destinations` for
phase 6. Each query has a differential test against Python's.
The native queue tests now run with MSC4354 on, so the loops look at the
sticky event backlog, and pin that it takes no GIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::catch_up` runs `_catch_up_transmission_loop` natively. It
reads the catch-up events and the room's forward extremities, loads
events through the shared event cache, filters the extremities with the
Rust `filter_events_for_server`, sends each room's PDUs through the
queue's sink, records the sticky event gap and moves the catch-up
position on. A destination catching up no longer takes the GIL, unless a
module wants to hear about delivered events.
`FederationStore` gains the catch-up queries, with differential tests
against Python's. The `catch_up` hook is given its queue, and a PDU can
now hold an event loaded in Rust.
Where Python would wait for a room's full state to filter, Rust sends
the original PDU, as it does for a room with partial state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`visibility::filter_events_for_server` reads the erased senders, the
partial state rooms, the history visibility and the target server's
memberships at each event through a `SyncReader`, and leaves the
per-event decision to `event_visible_to_server`. That now borrows its
arguments.
Python waits for the full state of an event whose memberships it needs.
Rust cannot, and returns `NeedsFullState` for the caller to handle.
The differential tests compare it with Python's on random rooms, an
outlier and a room with partial state, for every combination of flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Keep an unchanged device stream's position. The tokens are read after
the change check, so moving to them could skip a message that arrived
in between.
- Decode cross-signing keys as leniently as Python does, including
URL-safe base64.
- Mark the transaction open before awaiting BEGIN, so a dropped task
still rolls back.
- Tell modules about delivered events before the database writes, as
Python does.
- Update the stream change cache before its listeners, and log a
listener that fails.
- Check the metrics through a registry scrape, with a fresh event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`FederationStore` ports the store methods the federation transmission
loop calls around each transaction to the native Postgres driver:
`get_new_device_msgs_for_remote`, `delete_device_msgs_for_remote`,
`get_device_updates_by_remote` (with the cross-signing key updates and
cross-signatures), `mark_as_sent_devices_by_remote` and
`set_destination_last_successful_stream_ordering`. The stream change
cache checks those start with are left to the caller.
None of the writes has a Python cache to invalidate. The cross-signing
reads skip Python's caches.
`tests/storage/test_federation_store.py` runs each against the Python
method on one database, through a `FederationStoreForTest` pyclass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Stop catch-up before the next room once the loop is shut down. The Rust
loop cannot cancel the Python catch-up coroutine, and after a shutdown
the transaction manager returns without sending, so the room would be
recorded as sent.
- Record the request metrics when the Python fallback for reading a
transaction response fails.
- Give each transmission loop its own LoopActivity, so a loop that ends
cannot clear the parked flag of the next.
- Count the time spent reading a hook's result as GIL time of its site.
- Correct the AwaitableObserver::finished doc.
- Let test_send_receipts_with_backoff_sender accept either order of its
first two sends, which run at once with the queues in Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`FederationSender` makes a `RustPerDestinationQueue` for each
destination at the new `queues` level. It forwards the queue methods to
Rust and provides the hooks: catch-up, the sticky backlog, the device
EDUs, and recording a sent transaction (stream positions, the module
callback, the metrics).
The catch-up loop moves into `_CatchUpLoop`, which both queues share.
`StickyEventBacklogTracker.load_backlog` splits the database part out of
`prepare_transaction`.
The sender, catch-up and sticky event tests also run at this level, with
every reactor advance waiting for the Rust loops to settle.
`test_send_receipts_thread` is skipped there: it depends on coroutine
order, which the Rust loop changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Only build the Rust sender from a real `RustFederationClient`. A test
that mocks the federation client no longer fails with a `TypeError`.
- Log the `TX` and response lines under Python's logger names, so log
configs that name those loggers still apply.
- Move the PDUs and EDUs into the request body, so a transaction holds
one copy of them while it is retried, besides the canonical JSON.
- Drop the unused `time_now` from `Event::to_pdu_value`.
- Set the compare mode per `TransactionManager` in the tests that use
it, not on the class.
- Test that the tracing headers reach the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
At the new `transactions` level, `TransactionManager` hands each
transaction to the Rust sender. Python keeps the tracing span, the
module callback and the `last_sent_pdu_time` gauge.
In tests, every transaction body Rust builds is checked against the one
Python builds (`SYNAPSE_TEST_RUST_FED_STRICT`). The federation sender
and catch-up test cases also run at the new level, with Rust building
the transactions that their mocked `send_transaction` receives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Create the sleeper's `Notified` under the map lock, so a wake that lands
just before the sleep starts is not lost. Count sleeps per destination
and forget the destination in a drop guard, which also runs on cancel.
- Map an SRV `.` target to `ConnectError`, as Python's resolver does.
- Call a custom parser's `finish` when its last write fails, and when the
request is cancelled mid-read.
- Check the path is ASCII after the backoff check, as Python does.
- `put_json` takes `rewrite_age_ts`, which `send_transaction` sets. Any
other `json_data_callback` sends through the Python client.
- Count attempts in the no-retry test, and test the request counters,
the proxy-worker fallback, shutdown and the server-up wake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Add the `experimental_features.rust_federation_sending` level, `off` or
`client` for now. At `client`, `MatrixFederationHttpClient` sends its
JSON requests (`put_json`, `post_json`, `get_json`,
`get_json_with_headers`, `delete_json`, `send_direct_request`) through
the Rust client. Media downloads, and an instance that sends through a
proxy worker, stay on the Python path.
The facade keeps the tracing span around the call and injects its
headers. The request and response counters are bumped from the GIL hop
that returns the result. The limiter's background write is factored out
of `RetryDestinationLimiter` so both paths share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The Rust replacement for `MatrixFederationAgent` and its connection pool,
as a library: nothing in Synapse sends through it yet. The request layer
(signing, retries, backoff) comes next and will be its first caller.
`Agent::request` takes a `matrix-federation://` request, applies
`.well-known` delegation to the URI and the `Host` header, and sends it
through hyper-util's pooled client (5 idle connections per host, 120s
idle timeout, HTTP/1.1, no automatic retries), as Python's pool does.
The pool calls our connector, derived from matrix-hyper-federation-
client's `MatrixConnector` (Apache-2.0). It resolves the server name
(SRV), tries each server in order and returns the first error if all
fail. For each server it resolves addresses through the IP range policy,
connects TCP with Twisted's staggered attempts and 30s timeout, or
tunnels through `https_proxy` with `CONNECT` unless `no_proxy` exempts
the host, then does TLS verified against the delegated server name,
never the SRV target. The `.well-known` fetch shares the pool and, as in
Python, blocks IP literals by the blocklist alone.
Failures are a typed `TransportError` (DNS, connect, TLS, and so on), so
the request layer can tell a DNS failure from the rest. The agent
implements `FederationTransport`.
`RustFederationAgentForTest` exposes a blocking `request` to Python. The
trial tests drive it against a real TLS server on a thread, signed by
the test CA, covering the delegation, SRV, SNI, IP literal, blocklist,
certificate verification and connection reuse cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Add `rust/src/federation`, starting with the `X-Matrix` request
signature and the transport seam the request layer will send through.
The signing code comes from matrix-hyper-federation-client (Apache-2.0),
changed to use Synapse's canonical JSON serializer, Python's header
layout and quoting, and `destination_is` for identity servers. It keeps
that crate's signature test vectors. The body is serialized to canonical
JSON once, then both signed and sent in that form.
The homeserver's signing key crosses to Rust as its algorithm, version
and seed, taken from `hs.signing_key`. Rust never reads the key file.
`FederationTransport` is what the request layer will send through.
`PyTransport` implements it with a Python callable, which is how Rust
level tests will inject responses in place of `FakeTransport`.
A trial test checks the Rust header is byte-identical to the one
`MatrixFederationHttpClient.build_auth_headers` builds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The new `sync/relations.rs` ports the relations handler's
`get_bundled_aggregations`: thread summaries with participation and
ignored users' replies subtracted, references, and the latest edit. The
Rust sync handler bundles them for limited and newly joined timelines,
loads the redactions of the bundled events, and the encoder serializes
them. Python no longer bundles the rooms Rust computes, and the room
entries hand Python the bundles only when something reads the timeline.
A thread whose latest reply is from an ignored user falls back to
Python for that room: finding the latest reply that is not takes the
relations pagination, which is not ported.
The relations queries read the database each time. Python's relations
caches are not mirrored.
Shadow mode now compares the Rust bundles with Python's, in the encoded
rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The rooms the Rust sync handler computes now reach the servlet as
`RustJoinedSyncResult` and `RustArchivedSyncResult`. They subclass the
Python entries, and their fields become Python objects only when read.
The servlet encodes them in one call to the new Rust `RoomsEncoder`,
which ports `encode_room` over the existing serializer. If that call
fails, Python encodes them.
The Rust handler loads each room's redactions as it computes the room.
The serializer reads redactions, bundles and module additions through a
`SerializeLookups` trait, so that it takes Rust events as well as Python
maps. Bundled aggregations are still computed in Python.
Shadow mode now also compares the encoded JSON of each room.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`rust_sync: on` was refused unless `rust_sync_allow_unshared_event_cache`
was set, because the Rust sync handler held a second copy of every event
it served. Python and Rust now share one event cache, so the gate and its
option go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
`EventsWorkerStore._get_event_cache` is now the Rust `EventCache` that the
sync handler and state resolution read, so each event is held once.
The cache keeps Rust `CachedEvent` values. Python gets them through
`RustEventCache`, a `RustCache` subclass with the `AsyncLruCache` names the
rest of Synapse uses, and each hit wraps the cached event in a new `Event`
object. The wrapper shares the cached event's internal metadata, unsigned
data and signatures, so a change Python makes, such as adding
`prev_content` or flipping `outlier`, lands in the one copy the Rust
readers see. Python's `_event_ref` weak map is checked before the cache,
so while anything holds an event, a lookup returns that same object, as
before. A hit there marks the cache entry as read, and puts the event back
if it was evicted.
- The cache is sized from `caches.event_cache_size` and named
`*getEvent*`, so its metrics and `per_cache_factors` keep their names.
The Rust loader now counts its misses too.
- Invalidating an event drops its chain position as well, and a room
purge drops the room's events through the room index, plus the chain
cover and state group caches, as `RustStore.invalidate_event` and
`clear_event_cache` did. Those two calls are gone.
- `EventsWorkerStore` and `RustWorkerStore` run their `__init__` bodies
in a different order in `DataStore` and `GenericWorkerStore`, as they
sit differently in each MRO. So whichever runs first builds the
`RustStore`. A datastore without `RustWorkerStore`, such as the port
script's, gets a standalone `RustEventCache`.
- The external cache stubs (`get_external`, `set_external`,
`_invalidate_async_get_event_cache` and its room variant) are dropped.
They were documented no-ops.
- The sync shadow diff still ignores `prev_content` on state events.
Python keeps the objects it holds after the cache evicts them, so a Rust
reader can still load a second copy without it. Trial's cache of one
event makes that routine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
With `cache_autotuning` configured, `_expire_old_entries` walks Python's
global cache list from the coldest entry and evicts while jemalloc reports
too much memory allocated. Rust caches were not on that list, so they were
never evicted for memory, which matters once the event cache lives in Rust.
Each Rust cache's LRU order is already a sorted run, so the registry
merges them without a global list: `CacheRegistry::evict_coldest` keeps
evicting from the cache whose coldest entry is coldest, up to a bound and
a count. The Python walk calls it before evicting each Python entry, with
that entry's last access as the bound, so both sides go coldest first
together. It reports the coldest Rust entry left, so Python only calls
again once its walk passes that time. Memory is re-read after each batch
of 100 Rust evictions, as after 100 Python ones, and `min_cache_ttl` holds
for Rust entries too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
With `rust_sync: on`, Python hands the rooms section to the Rust handler in
two calls, fetching the sticky events of newly joined rooms between them.
Python computes the rooms the Rust handler leaves to it. If the Rust
handler fails to work out the room changes, Python computes the whole
section.
Shadow mode runs the Python and Rust sections at once. It compares the room
changes (newly joined and left rooms, invites and knocks) as well as each
room's entry.
Python still computes what feeds the steps before the rooms: the joined
rooms of `get_sync_result_builder`, ephemeral events, room account data,
tags and sticky events. It also drops the invites of ignored users and those
the invite rules ignore.
The unknown room version test also clears the Rust event cache, as the
restart it stands for would.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`on` serves each joined or archived room from the Rust handler. A room the
Rust handler cannot compute, or fails on, is computed by Python. Python's
fallback rooms share the Rust handler's lazy-loaded members caches, so a
room that moves between the two does not resend members.
`shadow` serves Python's entry and compares the Rust one with it, counting
the outcome in synapse_sync_rust_shadow_total. The two run at once, so the
response waits for the slower. Cancellation stops both.
Config parsing refuses `on` while the Rust and Python event caches are
separate, unless `rust_sync_allow_unshared_event_cache` is set. Under trial,
a Rust error or a shadow mismatch fails the test.
The call invite test now builds its events for the room's own version. An
event of another version is stored, but the Rust event loader cannot read
it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The queries of the per-room sync loop, on the native Postgres driver, each
a port of the store method it is named after. A `SyncReader` is made per
request and borrows a pooled connection for one group of statements at a
time, so that a sync worker with hundreds of requests in flight does not
pin a connection per room; a group that needs one snapshot runs in a
`REPEATABLE READ READ ONLY` transaction. State groups are loaded whole,
with the recursive CTE sent as one simple query per batch, and cached as
full maps.
The caches that copy a `@cached` function (rooms for user, room summary,
partial state rooms, erased users, retention policies, event to state
group, unread counts) live on the Rust store and are attached to the
Python cache each names by the new `RustWorkerStore`, which also takes
over building the Rust store. Unread counts are invalidated by room as
well as by room and user, as the Python tree cache is.
`tests/storage/test_sync_store.py` runs every query and the Python
method against the same database, with multi-writer tokens, redactions
of every kind, and the summary rotation of the unread counts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The sync store's caches are copies of `@cached` functions of the
datastore, and those are invalidated by several paths: a write on the
process, the caches stream, and the current state batch. Rather than tap
each path, a `DeferredCache` now takes mirrors, invalidated with its own
keys whenever it is, and `mirror_rust_cache` attaches a Rust cache as one,
passing the key as the tuple of arguments.
A `tree=True` cache is invalidated by a prefix of its key as well, so the
Rust cache gains `invalidate_matching`, which drops every entry whose key
satisfies a test, and a key type can decode a prefix into one. A key of
two strings, such as room and user, does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
`LazyLoadedMembersCache` as it is on develop after #20289: an LRU of the
members sent to a device, each stamped with the room position of the
response that carried it, and `start_request` forgetting the entries
recorded after the request's `since`. The per-device caches expire
after thirty minutes without use, as the Python `ExpiringCache` does.
The Rust sync handler holds them, not the store: they are per device and
no data change invalidates them. `SyncHandler.start_request` is its
first entry point.
`tests/sync/test_rust_ll_members.py` mirrors the Python cache's tests
and runs random operation sequences, with multi-writer positions,
through both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The per-event decision of `filter_and_transform_events_for_client`:
`_check_client_allowed_to_see_event` and the membership, history
visibility, send-to-client and effective visibility checks under it, the
call invite rule, and the membership annotation for MSC4115. The inputs
are the event, the user's membership and the history visibility at it as
events, and flags; loading those is the caller's job, so the module can
serve the sync handler and later ports alike. The erased-sender case
prunes through the existing Rust redaction.
The `allowed` closure in the Python is now a module-level function,
`_check_event_for_client`, with the same body, so that
`tests/sync/test_rust_visibility.py` can compare the two exactly on
random events, state and flags, sticky and outlier events included.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
`_calculate_state`, the `use_state_after` roll-back over state deltas of
the full-state path, and the delta loop of the incremental path. The
database reads that produce their inputs stay with the store.
The two Python loops were inline in the handler; they are now functions,
`_roll_back_state_over_deltas` and `_apply_current_state_deltas`, so that
`tests/sync/test_rust_state_delta.py` can compare them with the Rust on
random state maps and deltas. Behaviour is unchanged.
One deliberate difference: when the start and end of the timeline
disagree on a key that is in neither the timeline nor the previous state,
`_calculate_state` sends whichever event its set iteration visits last.
The Rust sends the one from the end of the timeline, the state the client
should end up with. The test allows the Python either.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
`Filter` and `FilterCollection` from `synapse.api.filtering`, built from
the filter JSON the Python one was built from, once per request. The
checks are pure functions of the filter and the event, over events,
presence updates and the JSON of ephemeral events and account data. The
one part that needs the database, the `related_by_*` relation checks, is
a trait the store will supply; `needs_relation_check` says when.
The config flags the Python reads (MSC3773, MSC3874, profile updates in
sync) are passed in, so the port has no reach into the config.
`tests/sync/test_rust_filter.py` compares the two on random filters and
random events, with and without the MSCs enabled.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The first of the pure parts of `/sync` that the Rust sync handler needs:
`RoomStreamToken` with its three string forms and its vector clock
arithmetic (`is_before_or_eq`, `copy_and_advance`, `bound_stream_token`),
`persisted_after`, and the pagination helpers of `stream.py` that turn
tokens into bounds and filter the rows that come back by writer. The SQL
is built in the Postgres row-comparison form only, as the handler reads
through the native Postgres driver.
Tokens cross the boundary as the Python class, converted both ways.
The multi-writer filtering is where a bug would be silent, so
`tests/sync/test_rust_stream_token.py` checks every operation against the
Python on random tokens with instance maps. `tests/sync/fixtures.py`
holds what the differential tests that follow share.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
`rust_sync` takes `off`, `shadow` or `on`, as `rust_state_res` does, and
`RustHandlers.sync` is built when the main database uses the native Rust
driver. The Python `SyncHandler` reads the option and takes the Rust
handler, warning and keeping the Python path when the driver is not the
Rust one. Nothing is served by Rust yet: the per-room entry points come
with the ports that follow.
Trial gets the same plumbing the state resolution has: `SYNAPSE_TEST_
RUST_SYNC` sets the option on every test homeserver, and the Rust handler
is wrapped so that each entry point's `_blocking` twin answers under the
fake reactor.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
Cancelling a deferred from `create_deferred` failed it at once and left
the tokio task running to completion, its result thrown away. That is
the right behaviour for a cache load, which other callers may be
waiting on, but wrong for work only the cancelled caller wanted: a
`/sync` request's per-room loop, say, which clients abandon constantly
and which holds a pooled connection while it runs.
`create_deferred_with` takes an `OnCancel`. With `Abort`, the deferred's
canceller aborts the task, so its future is dropped where it stands and
whatever it holds is released by `Drop`. The default stays as it was.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The first Rust database function to get a cache, and the first Python
`@cached` function to be replaced by one. The Rust `/versions` handler
calls `Store::is_feature_enabled_for_user` on every authenticated request,
and until now each call ran a query. The Python `is_feature_enabled` had a
cache of the same rows under another name, which every writer would have
had to invalidate alongside. Now there is one cache, keyed on
`(user_id, feature)`, and both sides read through it.
The main datastore builds the Rust `Store` in its `__init__`, over its own
`db_pool`, and `hs.get_rust_store()` hands it to the Rust handlers: the
datastore owns the store, the handlers borrow it. The store's
`is_feature_enabled_for_user` reaches Python as a callable cache object,
bound on the datastore under that name. Calling it answers a hit with an
already-fired Deferred, without leaving the reactor thread, and only a
miss goes through tokio. Being a `RustCache`, it is also what
`_invalidate_cache_and_stream`, replication and `register_cache` take, so
`set_features_for_user` invalidates it with the one call it already made,
and the name is spelled once in Rust, where the cache is built.
`is_feature_enabled` keeps its global-config short cut in Python and
delegates the per-user lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
