- Re-raise SlidingSyncUnknownPosition from finish_request. A racing
request on the same connection can clear the position before the
persist, and Python would fail the same way, so a Python retry only
repeats the work.
- Treat an empty events stream cache as a miss, as Python's
get_entities_changed does, so Rust asks the database.
- Copy the room statuses out of the connection state before building
Python objects, so the lock is never held across a Python call.
- Trace the rooms Python computes for Rust, as the Python path does.
- Test the per-room fallback: Python computes a room Rust leaves to it,
and its changes reach the Rust state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
In `rust_sliding_sync: on` mode, `current_sync_for_user` is now four Rust
calls around the Python extensions: load the connection state, compute
the interested rooms (the port of `_compute_interested_rooms_new_tables`),
compute the rooms, and finish the request (unsent and sent rooms, then
the persist).
The connection state is a Rust pyclass. The extensions read and update it
through views with the surface of Python's `PerConnectionState` and
`MutablePerConnectionState`. Receipts tokens stay in their database form
in Rust and become `MultiWriterStreamToken`s only when Python asks.
The whole request falls back to Python when the sliding sync tables are
not filled, for a filter Python does not implement either, and on any
Rust error. Rooms Rust leaves to Python run `get_room_sync_data` over a
Python copy of their slice of the state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
SlidingSyncHandler.generate_room_entries runs get_room_sync_data for a
batch of rooms, twenty at a time, and returns each room's entry with the
two changes it makes to the connection state. encode_rooms is the
servlet's encode_rooms for those entries.
A room's state loads reuse the events its timeline loaded, as Python's
held-event map does, so state events carry the same prev_content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
filter_events_for_client takes is_peeking, which v2 passes as false and
sliding sync sets for rooms the user is not joined to. Events gain
instance_name, and unsigned data gains invite_room_state and
knock_room_state accessors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`max_event_pos` now takes `get_max_pos_of_last_change` for the rooms the
events stream cache knows, as `_bulk_get_max_event_pos` does. The port
keeps Python's quirk: the cache answers only when it knows every missing
room, because Python's batch loop goes over all of them.
The membership change tests now add delta rows without an event, as the
server leaving a room or a state reset write them. The branches that
handle those rows were not reached before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`SlidingSyncReader` gains the reads and writes of a connection's
per-request state:
- `connection_lazy_members`: `get_sliding_sync_connection_lazy_members`
- `load_connection_state`: the `get_and_clear_connection_positions`
transaction. It bumps `last_used_ts`, deletes the other positions,
un-forks the lazy members, drops the unused required state rows and
reads the state.
- `persist_connection_state`: the `persist_per_connection_state`
transaction, from the row lock through the lazy member upserts, with
`UNNEST` for the batched upserts and deletes
The state crosses as `StoredConnectionState` and `ConnectionStateUpdates`,
with each stream's tokens as strings, as `PerConnectionStateDB` holds
them. Parsing the tokens is the caller's job once the state is Rust-owned.
An unknown position is the typed `UnknownPosition`, which the test entry
points raise as `SlidingSyncUnknownPosition`.
Both run as `REPEATABLE READ` read-write transactions, retried up to
five times on a serialization failure or a deadlock, as `runInteraction`
does. The caller passes the time in, so the tests drive both sides from
one clock. Loaded states are memoized by their arguments in a cache
named after Python's `@cached` one, which nothing invalidates. A
long-poll's second load and a client's retry therefore run the
transaction once.
Rooms share required state rows by their JSON. `encode_required_state`
writes it as Python's `json_encoder` does, ASCII-escaped, so that rows
written by either side dedup.
The test runs twelve random operations through Python on one connection
and through Rust on another, 20 seeds. The operations are persists on
the latest position, on a fork or from scratch, and loads of current,
stale and foreign positions. After each step it compares the two
connections' rows (positions by order, required state IDs by their
JSON) and what each load returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/storage/sliding_sync_store.rs` adds `SlidingSyncReader`. It is
a `SyncReader` with the sliding sync queries on top, so one request
counts its v2 and sliding sync reads in one set of stats. Only the path
Python takes once the background updates have finished is ported.
- membership snapshots: `rooms_from_snapshots`, `self_leave_rooms_after`,
`snapshot_room` and `snapshot_rooms`
- membership changes: `snapshot_membership_changes` and
`membership_delta_changes`. The caller passes the membership stream
cache's answer in as a bool.
- stream positions: `rooms_with_updates_since`, which takes the events
stream cache's changed rooms when it has them, and `max_event_pos` and
`last_event_pos_before`
- room metadata: `room_type`, `room_version_id`, `partial_rooms`,
`tags_for_user`, `cached_member_counts` and `is_encrypted_at`
Each Python `@cached` method behind these gets a Rust cache that mirrors
it by name: the snapshots, `_get_max_event_pos`, `get_member_counts`,
`get_room_type`, `get_room_version_id`, `get_partial_rooms`,
`get_tags_for_user` and `get_global_account_data_by_type_for_user`. The
v2 reader's global account data read now goes through that last one.
`get_partial_rooms` takes no arguments, so `()` becomes a cache key.
A row's null instance name reads as "master", as everywhere else in
Rust. Python keeps the `None` in `get_sliding_sync_membership_changes`.
That only matters for a token whose instance map names "master".
`tests/storage/test_sliding_sync_store.py` runs each query against the
Python method on the same database, with the writers scattered over
four instances for the token-bounded reads.
Python's `get_rooms_that_have_updates_since_sliding_sync_table` calls
`have_finished_sliding_sync_background_jobs()` without awaiting it. The
coroutine is always truthy, so Python never takes its "assume all rooms
changed" branch. The Rust port does not have that branch either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/extensions/relevant.rs` holds the matching of an
extension's `lists` and `rooms` against the response. Phase 5 uses it,
and it is cheap to port now. A list without exactly one `SYNC`
operation fails, as Python's assertion does: that is a list with no
ranges or several, under an extension that names it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room.rs` holds the parts of `get_room_sync_data`
that make no reads:
- the timeline bounds (`from_bound`, `initial`, the expanded timeline)
- `num_live`
- the stripped state of an invite or knock
- the string fields of the name, avatar and member events, and the
hero assembly
- `_get_bump_stamp` split around its two reads, plus the
`always_return_bump_stamp` rule and the clamp to zero
- the room config the connection records afterwards
Three of these move, unchanged, out of `get_room_sync_data` into Python
helpers, so the tests compare against the real code: `_timeline_bounds`,
`_stripped_state` and `_room_config_to_persist`. `_get_bump_stamp` is
tested as it is, with a stand-in store.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/sort.rs` holds `sort_rooms` as a
`SortPlan`: the decision logic with the stream change cache shortcut,
and the caller fetching what each step asks for. `lists.rs` holds the
ranges to `SYNC` operations with the full-range no-sort shortcut, the
config combining across lists and room subscriptions, and the decision
half of `_filter_relevant_rooms_to_send`, including the reset for a
connection with too many rooms to update.
The Python ranges loop moves, unchanged, into `_sync_list_ops`, so the
test can call it. It keeps a quirk the Rust ports: once a room combines
with a config from an earlier list, every later room in the same list
takes the combined config too.
The relevant room map is an `IndexMap`, which keeps Python's dict order,
so `indexmap` becomes a direct dependency. It was already in the tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/filter.rs` holds
`filter_membership_for_sync` and `filter_rooms_using_tables` over
`RoomForUser`, the Rust form of `RoomsForUserSlidingSync`. The caller
passes the DM rooms, the user's tags and whether the connection sent a
room before, which decides the state-reset keep rule. The `spaces` and
`room_name_like` filters return an error, as Python raises
`NotImplementedError` for them.
The filtered rooms keep their input order. Python's are in set order.
The differential test calls Python's method with a stand-in store over
2000 generated room sets and filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/connection.rs` holds `HaveSentRoom`,
`RoomStatusMap` and its mutable twin with the live-only transitions,
`PerConnectionState` and `MutablePerConnectionState`, and
`RoomLazyMembershipChanges` with the hourly last-seen rewrite. Python's
`ChainMap` becomes the previous maps behind an `Arc` plus the request's
updates, so persisting can write only the updates, as it does today.
The functions take the time as an argument. Phase 4 passes
`runtime.clock()`, which follows trial's virtual time. The receipts
stream's token type stays a parameter until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/required_state.rs` holds `_required_state_changes`
and the derivation in `get_room_sync_data` of the state filter and the
member sets from a room's `required_state` and timeline. The Python
derivation moves, unchanged, into two module-level helpers,
`_required_state_request` and `_state_filter_with_meta`, so the
differential tests can call it.
Python's results depend on set iteration order in three places, and Rust
takes one of Python's possible outcomes in each:
- The previous state keys kept over the 100-key cap: Python slices a
set, Rust takes the first in sorted order.
- Converting explicit members to lazy ones reads a loop variable that
leaked out of the loops before it. Without state deltas, Python's is
the last type of a set union in hash order, so whether those members
join the lazy cache varies between processes. Rust reads
`m.room.member`'s.
- `must_await_full_state` raises on an empty state key under the `*`
type, unless a remote user ID comes first in the set. Rust checks the
user IDs first.
The tests compare over 3000 generated inputs each, and allow exactly
these differences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/config.rs` holds `from_room_config`, `combine` and
`must_await_full_state`, the wildcard collapsing rules of a list's or a
room subscription's `required_state`. A differential test compares each
with the Python over 2000 generated configs.
`must_await_full_state` fails where Python raises: an empty state key
under the `*` type, which Python indexes into (`""[0]`).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Four reads on `SyncReader` for the per-room step of a later phase, each
checked against the Python store method in `tests/storage/test_sync_store.py`:
- `state_deltas_by_stream_id`: `get_current_state_deltas_for_room`, bounded
on the delta rows' own `stream_id`. Sliding sync reads this variant, not
the by-event-position one the v2 port uses.
- `last_event_of_types_before`: `get_last_event_pos_in_room_before_stream_ordering`
with `event_types`, for the bump stamp. `last_event_before` calls it with
no types.
- `member_counts`: `get_member_counts`.
- `latest_bump_stamp`: `get_latest_bump_stamp_for_room`.
None is cached yet. The plan mirrors `get_member_counts` by name in phase 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Python validates the body with pydantic as before. Rust parses the same
bytes with serde into `SlidingSyncBody`, which follows the pydantic model
field for field, including where it tells an explicit `null` from a missing
field. Lists and room subscriptions keep document order, and a repeated key
keeps its first position and last value, as the dict `json.loads` builds.
`SlidingSyncRequest` is the body plus what Python adds: the user and device,
the room key and connection position of `pos`, the room key and string suffix
of the token the response runs up to, the admin's soft-failed policy, whether
the background updates have finished, the excluded rooms and the extension
flags. `SlidingSyncHandler._build_rust_request` builds it.
The servlet keeps the body's bytes in `SlidingSyncConfig.body_bytes` when the
Rust handler is in use. A config built without a request is re-encoded.
`tests/rest/client/sliding_sync/test_rust_request.py` parses 400 generated
bodies (unknown fields, nulls, repeated keys, escapes) both ways and
compares, and checks the re-encoding parses the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Phase 0 of the sliding sync port (renders/sliding-sync-rust-plan.md). No
behaviour change: nothing calls the Rust handler yet.
- `experimental_features.rust_sliding_sync: off | on`. There is no shadow
mode. Booleans are accepted, as for `rust_sync`.
- `synapse_rust.sliding_sync.SlidingSyncHandler`, built by `RustHandlers`
on the native driver only. It shares the v2 sync handler's `SyncStore`
and carries the pool, deferred and blocking helpers the later phases'
entry points use.
- `SlidingSyncStats`, observed per request into the
`synapse_sliding_sync_rust_*` histograms. `SyncDbStats` now counts the
transactions a reader begins.
- The `synapse_sliding_sync_rust_rooms_total` and
`synapse_sliding_sync_rust_requests_total` counters.
- `SYNAPSE_TEST_RUST_SLIDING_SYNC` and `_STRICT` for trial, with the handler
wrapped in `_BlockingRustCalls`, and `RUST_SLIDING_SYNC` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
With the main database on the native Postgres driver, every event load
goes through the Rust `EventLoader`. `EventsWorkerStore` sends the events
its own lookups miss to `RustStore.load_events`, and `_get_events_from_db`
to `reload_events`. Loads from Python, sync and state resolution then
coalesce in the shared event cache, and an invalidation during a load
stops it caching a stale entry. Python's fetch threads and in-flight map
stay only for other databases.
- `EventLoader::load` goes through `Cache::get_or_load_many`. The extra
redaction events it reads are returned but no longer cached.
- `PoolQuerier` checks its connection out before claiming any loads, so
readers that hold connections while they wait cannot starve a load.
- An unusable event raises `InvalidEventError` or
`DatabaseCorruptionError`, as from Python, which some callers catch.
- Cancelling a Python caller leaves the load running for the others, and
`evt_db_fetch_count` counts only what a caller read itself.
- `RustStore` holds the `DatabasePool` weakly and reads its pool per call.
Tests: trial makes the loads block, as it does Python's transactions
(`_BlockingRustCalls`, renamed from `_BlockingRustSyncHandler`). The
cancellation and outage tests get Rust variants. Two tests stop depending
on how far requests advance the clock or on which `DatabasePool` object
a worker store keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
`Cache::get_or_load_many` looks up several keys at once and loads the
misses with one call. Each key coalesces with loads already in flight,
single or batched, and a key invalidated mid-load is answered but not
cached. Waiters now get `None` for a key a batch load did not find, and
a single load waiting on such a key loads it itself.
`Cache::abandon_loads` stops every in-flight load from caching, for an
invalidation that cannot name the keys it affects. A room purge uses it
on the event cache, as the room of an event being loaded isn't known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
- The server filter read each restricted event's state group before it
checked for partial state, and through a cache that a prefill never
reached. After an event got its full state, a retry could use the old
partial group. It now checks first, then reads those groups from the
database. A prefill of a Python cache now invalidates its Rust mirrors.
- Catch-up ended in two steps, so a PDU that send_pdu skipped in between
was lost until the next catch-up. One lock now covers both.
- The sticky backlog query reads a fresh snapshot per statement, so
events could expire between its cleanup and its retry. It now cleans
up again, a few times, before giving up.
- Tests for each, plus catch-up's batch limit, a failed catch-up send,
module notification, a filtered-out sticky batch, the hand-off to
Python, the backlog cleanup boundary and paging outstanding
destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::sticky_backlog` loads a backlogged batch, cleaning up rooms
whose sticky events expired, reads the events through the shared event
cache and filters them for the destination. Its completion records the
batch natively. Where the filter would wait for full state, the batch is
left to Python, as before.
`FederationStore` also gains `get_catch_up_outstanding_destinations` for
phase 6. Each query has a differential test against Python's.
The native queue tests now run with MSC4354 on, so the loops look at the
sticky event backlog, and pin that it takes no GIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`NativeHooks::catch_up` runs `_catch_up_transmission_loop` natively. It
reads the catch-up events and the room's forward extremities, loads
events through the shared event cache, filters the extremities with the
Rust `filter_events_for_server`, sends each room's PDUs through the
queue's sink, records the sticky event gap and moves the catch-up
position on. A destination catching up no longer takes the GIL, unless a
module wants to hear about delivered events.
`FederationStore` gains the catch-up queries, with differential tests
against Python's. The `catch_up` hook is given its queue, and a PDU can
now hold an event loaded in Rust.
Where Python would wait for a room's full state to filter, Rust sends
the original PDU, as it does for a room with partial state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`visibility::filter_events_for_server` reads the erased senders, the
partial state rooms, the history visibility and the target server's
memberships at each event through a `SyncReader`, and leaves the
per-event decision to `event_visible_to_server`. That now borrows its
arguments.
Python waits for the full state of an event whose memberships it needs.
Rust cannot, and returns `NeedsFullState` for the caller to handle.
The differential tests compare it with Python's on random rooms, an
outlier and a room with partial state, for every combination of flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Keep an unchanged device stream's position. The tokens are read after
the change check, so moving to them could skip a message that arrived
in between.
- Decode cross-signing keys as leniently as Python does, including
URL-safe base64.
- Mark the transaction open before awaiting BEGIN, so a dropped task
still rolls back.
- Tell modules about delivered events before the database writes, as
Python does.
- Update the stream change cache before its listeners, and log a
listener that fails.
- Check the metrics through a registry scrape, with a fresh event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`FederationStore` ports the store methods the federation transmission
loop calls around each transaction to the native Postgres driver:
`get_new_device_msgs_for_remote`, `delete_device_msgs_for_remote`,
`get_device_updates_by_remote` (with the cross-signing key updates and
cross-signatures), `mark_as_sent_devices_by_remote` and
`set_destination_last_successful_stream_ordering`. The stream change
cache checks those start with are left to the caller.
None of the writes has a Python cache to invalidate. The cross-signing
reads skip Python's caches.
`tests/storage/test_federation_store.py` runs each against the Python
method on one database, through a `FederationStoreForTest` pyclass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Stop catch-up before the next room once the loop is shut down. The Rust
loop cannot cancel the Python catch-up coroutine, and after a shutdown
the transaction manager returns without sending, so the room would be
recorded as sent.
- Record the request metrics when the Python fallback for reading a
transaction response fails.
- Give each transmission loop its own LoopActivity, so a loop that ends
cannot clear the parked flag of the next.
- Count the time spent reading a hook's result as GIL time of its site.
- Correct the AwaitableObserver::finished doc.
- Let test_send_receipts_with_backoff_sender accept either order of its
first two sends, which run at once with the queues in Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`DestinationQueue` holds what waits to go to one destination: PDUs,
EDUs, keyed EDUs, presence and receipts, with Python's merge rules and
limits. The first thing queued starts its transmission loop as a tokio
task, which builds and sends transactions until there is nothing left,
as `_transaction_transmission_loop` does.
Catch-up, the sticky event backlog and the loop's database work stay
Python for now, behind `QueueHooks`. `RustDestinationQueue` is the
pyclass the Python facade wraps, and calls those hooks on it.
`run_python_awaitable_observed` converts a Python result on the reactor
and tells an observer when the awaitable starts and finishes. The loop
uses it so tests can tell a loop waiting on the reactor from one with
work to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Only build the Rust sender from a real `RustFederationClient`. A test
that mocks the federation client no longer fails with a `TypeError`.
- Log the `TX` and response lines under Python's logger names, so log
configs that name those loggers still apply.
- Move the PDUs and EDUs into the request body, so a transaction holds
one copy of them while it is retried, besides the canonical JSON.
- Drop the unused `time_now` from `Event::to_pdu_value`.
- Set the compare mode per `TransactionManager` in the tests that use
it, not on the class.
- Test that the tracing headers reach the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`TransactionSender` builds `/send` transactions and sends them through
the Rust federation client, with the options `send_transaction` uses.
It reads the response and returns the ids of the PDUs the remote
accepted. `Event::to_pdu_value` builds the PDU JSON from the Rust event
without the GIL.
`RustTransactionManager` is the Python facade. It takes the events and
EDU dicts, and returns a Deferred. A response body that serde_json
refuses goes to Python's decoder, as `JsonParser` would.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
- Create the sleeper's `Notified` under the map lock, so a wake that lands
just before the sleep starts is not lost. Count sleeps per destination
and forget the destination in a drop guard, which also runs on cancel.
- Map an SRV `.` target to `ConnectError`, as Python's resolver does.
- Call a custom parser's `finish` when its last write fails, and when the
request is cancelled mid-read.
- Check the path is ASCII after the backoff check, as Python does.
- `put_json` takes `rewrite_age_ts`, which `send_transaction` sets. Any
other `json_data_callback` sends through the Python client.
- Count attempts in the no-retry test, and test the request counters,
the proxy-worker fallback, shutdown and the server-up wake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Port `MatrixFederationHttpClient._send_request` and `_handle_response`
to Rust, over the phase 1 agent: destination validation and the
whitelist, the retry limiter, `X-Matrix` signing per attempt, the long
and short retry schedules with jitter, `wake_destination`, timeouts on
the virtual clock, the trailing slash retry, and the mapping of each
failure to the exception Python raises.
The JSON parsers run on tokio without the GIL. Python's decoder is the
fallback for the JSON serde_json refuses, such as lone surrogates. A
custom `ByteParser` is fed under the GIL in 64 KiB chunks. The
transaction manager's `json_data_callback` becomes a `rewrite_age_ts`
option, so a retry needs no call into Python.
The retry timings are read through a Rust mirror of
`get_destination_retry_timings`. Writes go to a Python callable, which
invalidates the cache across workers and reports a server coming back.
Every GIL acquisition in the federation code goes through `with_gil`,
which counts calls and hold time per call site.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
A shared .well-known lookup only ran while a caller awaited it. If every
caller gave up, the next request resumed it with its fetch timeout long
expired, and cached a failure for an hour. Run it as its own task.
The TLS handshake and a proxy's CONNECT answer had no timeout. hyper
finishes some connections in a background task that no request timeout
reaches, so a peer that stalled after accepting leaked the task and its
socket. Give each 30 seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The Rust replacement for `MatrixFederationAgent` and its connection pool,
as a library: nothing in Synapse sends through it yet. The request layer
(signing, retries, backoff) comes next and will be its first caller.
`Agent::request` takes a `matrix-federation://` request, applies
`.well-known` delegation to the URI and the `Host` header, and sends it
through hyper-util's pooled client (5 idle connections per host, 120s
idle timeout, HTTP/1.1, no automatic retries), as Python's pool does.
The pool calls our connector, derived from matrix-hyper-federation-
client's `MatrixConnector` (Apache-2.0). It resolves the server name
(SRV), tries each server in order and returns the first error if all
fail. For each server it resolves addresses through the IP range policy,
connects TCP with Twisted's staggered attempts and 30s timeout, or
tunnels through `https_proxy` with `CONNECT` unless `no_proxy` exempts
the host, then does TLS verified against the delegated server name,
never the SRV target. The `.well-known` fetch shares the pool and, as in
Python, blocks IP literals by the blocklist alone.
Failures are a typed `TransportError` (DNS, connect, TLS, and so on), so
the request layer can tell a DNS failure from the rest. The agent
implements `FederationTransport`.
`RustFederationAgentForTest` exposes a blocking `request` to Python. The
trial tests drive it against a real TLS server on a thread, signed by
the test CA, covering the delegation, SRV, SNI, IP literal, blocklist,
certificate verification and connection reuse cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The parts of the outbound federation agent that decide where to connect
and what to trust, each a port of the Python it replaces, with Rust unit
tests. Nothing uses them yet.
- `resolve`: `_matrix-fed._tcp`, then `_matrix._tcp`, then port 8448, as
`MatrixHostnameEndpoint._resolve_server`. SRV records are ordered as
`_sort_server_list` (RFC 2782), cached while every record is live, and
the stale answer is used when a refetch fails. A lone `.` target means
the service is unavailable. SRV lookups go through hickory, and A/AAAA
through the system resolver, as Twisted uses getaddrinfo. Parts derive
from matrix-hyper-federation-client (Apache-2.0).
- `well_known`: `WellKnownResolver`, with Python's cache periods, jitter,
grace period, had-valid memory, size cap and retries, behind a
`WellKnownFetcher` trait so the state machine is tested without HTTP.
Two additions: each fetch attempt has a timeout (`client_timeout`), and
concurrent lookups for one server name share a fetch. An `m.server`
that is not a valid URI authority now counts as an invalid response.
- `policy`: the IP range lists, refusing a name if any of its addresses
is blocked (the DNS rebinding defence), and the `https_proxy`
settings with Python's `no_proxy` matching.
- `tls`: `FederationPolicyForHTTPS` on rustls: platform roots or the
custom CA list, verification off (still sending SNI), the per-host
verification whitelist. rustls has no TLS 1.0 or 1.1, so a minimum
below 1.2 means 1.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
A dropped Clock::sleep left its waker in VIRTUAL_SLEEPERS until the
virtual time passed its deadline, which a later test case's reactor may
never do. Prune closed wakers when sleeping and when the time moves.
Nothing constructs IpBlocked: a blocked address is a failed DNS lookup,
as in Python.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Extend the Rust view of the homeserver config with what the federation
request layer and agent need:
- `config.federation`: timeouts, retry counts and delays, destination
backoff, the domain whitelist and the metrics domains.
- `config.tls`: certificate verification, the per-host verification
whitelist, the minimum TLS version, and the custom CA list. Python now
keeps the PEM text of that list too, so Rust never reads the files.
- `config.server`: the federation IP range block and allow lists (taken
from `netaddr.IPSet`s), the user agent suffix and the proxy config.
The verification whitelist holds Python regexes made by
`glob_to_regex`. They translate to Rust's syntax with two changes: `\Z`
becomes `\z` and `re.IGNORECASE` becomes `(?i)`.
The changelog entry needs renaming to the PR number.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Add `rust/src/federation`, starting with the `X-Matrix` request
signature and the transport seam the request layer will send through.
The signing code comes from matrix-hyper-federation-client (Apache-2.0),
changed to use Synapse's canonical JSON serializer, Python's header
layout and quoting, and `destination_is` for identity servers. It keeps
that crate's signature test vectors. The body is serialized to canonical
JSON once, then both signed and sent in that form.
The homeserver's signing key crosses to Rust as its algorithm, version
and seed, taken from `hs.signing_key`. Rust never reads the key file.
`FederationTransport` is what the request layer will send through.
`PyTransport` implements it with a Python callable, which is how Rust
level tests will inject responses in place of `FakeTransport`.
A trial test checks the Rust header is byte-identical to the one
`MatrixFederationHttpClient.build_auth_headers` builds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Tokio's timers run on real time, but trial drives a virtual reactor clock
that only moves when a test advances it. A Rust task that sleeps with
`tokio::time::sleep` under trial waits out real time while the test
believes none has passed.
`Clock::sleep` sleeps on the virtual clock while a test has pinned it:
`set_virtual_time_msec` wakes the sleepers that are due. Otherwise it is
`tokio::time::sleep`. Code that sleeps takes a `Sleeper`, so that its
Rust unit tests can use tokio's pausable clock instead, since the
virtual time is process-global and an earlier test may have pinned it.
The federation sender port needs this for its backoff and well-known
retry sleeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The new `sync/relations.rs` ports the relations handler's
`get_bundled_aggregations`: thread summaries with participation and
ignored users' replies subtracted, references, and the latest edit. The
Rust sync handler bundles them for limited and newly joined timelines,
loads the redactions of the bundled events, and the encoder serializes
them. Python no longer bundles the rooms Rust computes, and the room
entries hand Python the bundles only when something reads the timeline.
A thread whose latest reply is from an ignored user falls back to
Python for that room: finding the latest reply that is not takes the
relations pagination, which is not ported.
The relations queries read the database each time. Python's relations
caches are not mirrored.
Shadow mode now compares the Rust bundles with Python's, in the encoded
rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The rooms the Rust sync handler computes now reach the servlet as
`RustJoinedSyncResult` and `RustArchivedSyncResult`. They subclass the
Python entries, and their fields become Python objects only when read.
The servlet encodes them in one call to the new Rust `RoomsEncoder`,
which ports `encode_room` over the existing serializer. If that call
fails, Python encodes them.
The Rust handler loads each room's redactions as it computes the room.
The serializer reads redactions, bundles and module additions through a
`SerializeLookups` trait, so that it takes Rust events as well as Python
maps. Bundled aggregations are still computed in Python.
Shadow mode now also compares the encoded JSON of each room.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
`EventsWorkerStore._get_event_cache` is now the Rust `EventCache` that the
sync handler and state resolution read, so each event is held once.
The cache keeps Rust `CachedEvent` values. Python gets them through
`RustEventCache`, a `RustCache` subclass with the `AsyncLruCache` names the
rest of Synapse uses, and each hit wraps the cached event in a new `Event`
object. The wrapper shares the cached event's internal metadata, unsigned
data and signatures, so a change Python makes, such as adding
`prev_content` or flipping `outlier`, lands in the one copy the Rust
readers see. Python's `_event_ref` weak map is checked before the cache,
so while anything holds an event, a lookup returns that same object, as
before. A hit there marks the cache entry as read, and puts the event back
if it was evicted.
- The cache is sized from `caches.event_cache_size` and named
`*getEvent*`, so its metrics and `per_cache_factors` keep their names.
The Rust loader now counts its misses too.
- Invalidating an event drops its chain position as well, and a room
purge drops the room's events through the room index, plus the chain
cover and state group caches, as `RustStore.invalidate_event` and
`clear_event_cache` did. Those two calls are gone.
- `EventsWorkerStore` and `RustWorkerStore` run their `__init__` bodies
in a different order in `DataStore` and `GenericWorkerStore`, as they
sit differently in each MRO. So whichever runs first builds the
`RustStore`. A datastore without `RustWorkerStore`, such as the port
script's, gets a standalone `RustEventCache`.
- The external cache stubs (`get_external`, `set_external`,
`_invalidate_async_get_event_cache` and its room variant) are dropped.
They were documented no-ops.
- The sync shadow diff still ignores `prev_content` on state events.
Python keeps the objects it holds after the cache evicts them, so a Rust
reader can still load a second copy without it. Trial's cache of one
event makes that routine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
With `cache_autotuning` configured, `_expire_old_entries` walks Python's
global cache list from the coldest entry and evicts while jemalloc reports
too much memory allocated. Rust caches were not on that list, so they were
never evicted for memory, which matters once the event cache lives in Rust.
Each Rust cache's LRU order is already a sorted run, so the registry
merges them without a global list: `CacheRegistry::evict_coldest` keeps
evicting from the cache whose coldest entry is coldest, up to a bound and
a count. The Python walk calls it before evicting each Python entry, with
that entry's last access as the bound, so both sides go coldest first
together. It reports the coldest Rust entry left, so Python only calls
again once its walk passes that time. Memory is re-read after each batch
of 100 Rust evictions, as after 100 Python ones, and `min_cache_ttl` holds
for Rust entries too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
Preparation for making the Rust event cache the one `EventsWorkerStore`
reads through.
- `Cache::new_indexed` keeps the keys grouped by a value-derived index key,
maintained on every insert and removal path, and `invalidate_index` drops
one group. A room purge invalidates the event cache by room, and
retention runs that for every room with a policy, so it must not walk
the whole cache.
- `Cache::touch` marks an entry as read without cloning it,
`get_or_count_miss` and `get_uncounted` cover the metrics cases of
Python's `update_metrics`, and `count_miss` records a miss found
elsewhere.
- `CachedEvent` works out its auth event IDs on first use. An event filled
from Python is usually never read by state resolution, so it should not
pay to parse them. `auth_event_ids` is fallible now, and its callers in
state resolution propagate the error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
Phase 4 of the sync-in-Rust plan. `SyncHandler.room_changes` ports
`_get_room_changes_for_incremental_sync` and
`_get_room_changes_for_initial_sync`: which rooms the response reports, and
which the user joined or left since the last sync. It fetches the timelines
of the joined rooms itself, as `get_room_events_stream_for_rooms` does.
`SyncHandler.generate_room_entries` then computes every joined and archived
room's entry, ten at a time. A room it cannot compute, or fails on, comes
back in the result for Python to compute. A panic in a room is that room's
error, so it cannot fail the rooms that have recorded their lazy-loaded
members.
The `state_after` rollback now reads the state deltas with no upper bound,
after the current state, so that it covers every change the current state
may hold.
These replace the per-room `generate_room_entry` and `RoomSyncInput`. The
store gains `event_position`, for the leave events of archived rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
Phase 3 of the sync-in-Rust plan: `SyncHandler.generate_room_entry` ports
`_generate_room_entry` for a joined or archived room, reading through the
sync store. It covers the timeline (gap check, filtering, visibility and the
backward pagination loop), the state delta on the full, incremental and
`state_after` paths with the lazy-loaded members cache, the room summary,
sticky events and unread counts. Partial state rooms raise
`RustSyncFallback`. Bundled aggregations stay with Python.
A room records the lazy-loaded members it sends only once its entry is
computed. A room that fails part way records none, so Python can compute it
from the same cache. Python reads and writes that cache through
`ll_member_was_sent` and `ll_member_mark_sent`.
The visibility checks get a batch loader over the store, and the store gains
`events_have_relations` for relation filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The queries of the per-room sync loop, on the native Postgres driver, each
a port of the store method it is named after. A `SyncReader` is made per
request and borrows a pooled connection for one group of statements at a
time, so that a sync worker with hundreds of requests in flight does not
pin a connection per room; a group that needs one snapshot runs in a
`REPEATABLE READ READ ONLY` transaction. State groups are loaded whole,
with the recursive CTE sent as one simple query per batch, and cached as
full maps.
The caches that copy a `@cached` function (rooms for user, room summary,
partial state rooms, erased users, retention policies, event to state
group, unread counts) live on the Rust store and are attached to the
Python cache each names by the new `RustWorkerStore`, which also takes
over building the Rust store. Unread counts are invalidated by room as
well as by room and user, as the Python tree cache is.
`tests/storage/test_sync_store.py` runs every query and the Python
method against the same database, with multi-writer tokens, redactions
of every kind, and the summary rotation of the unread counts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The sync store's caches are copies of `@cached` functions of the
datastore, and those are invalidated by several paths: a write on the
process, the caches stream, and the current state batch. Rather than tap
each path, a `DeferredCache` now takes mirrors, invalidated with its own
keys whenever it is, and `mirror_rust_cache` attaches a Rust cache as one,
passing the key as the tuple of arguments.
A `tree=True` cache is invalidated by a prefix of its key as well, so the
Rust cache gains `invalidate_matching`, which drops every entry whose key
satisfies a test, and a key type can decode a prefix into one. A key of
two strings, such as room and user, does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The state res store built events from rows itself, unredacted. Sync sends
events to clients, so it needs the whole of `_get_events_from_db`: the
`redactions` and `room_ban_redactions` tables, the recheck of redactions
whose authority was not verified at persist time, and `get_events_as_list`
on top, with its redact behaviours and the previous content of state
events.
`storage/events.rs` holds that port, over a `Querier` so it runs inside
the state res transaction as well as on a borrowed sync connection. The
event cache entry becomes a `CachedEvent`: the event as persisted and its
redacted form, as Python's `EventCacheEntry`. `StateResEvent` is an alias
of it, and the state res store loads through the shared code.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
