Behind `experimental_features.rust_room_creation: on`, the room handler
collects the events that follow the create event (the creator's join,
power levels, join rules, history visibility, the initial state, name,
topic) as `(type, state_key, content)`, and `LocalEventSigner` builds,
hashes and signs them all in one call. Each event's prev event, depth
and auth events follow from the one before, as `create_event` gives
them for a batch.
Python keeps every check: the requester checks ran when the create
event was built, and each built event still goes through
`validate_builder`, `validate_new` and the relation check before
joining the batch, which persists through `handle_new_client_event` as
before. Invites are still built by `create_event`.
Rooms the Rust builder does not cover are built in Python and counted
as fallbacks: v1 and v2 rooms, MSC4242 state DAG rooms, appservice
senders, modules with `check_event_allowed`, initial `m.room.aliases`
events, and `m.call.invite` or `m.room.create` in the initial state. If
Rust fails, the events are built in Python. If Python rejects them too,
the request was bad and gets Python's error. Otherwise it counts as a
Rust error. In tests,
`SYNAPSE_TEST_RUST_ROOM_CREATION=on` turns the builder on and makes a
failure fail the test.
Python orders auth events by iterating a set, so the order changes with
the process's hash seed. Rust keeps the order of the auth types. A test
creates rooms of several configurations and room versions in Python,
rebuilds their events in Rust from the same inputs, and compares them
with the auth events taken as a set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Rust could read state groups but not write them. A new
`store_state_group_chain` on the `RustStore` writes the chain
`store_state_deltas_for_batched` stores: an optional full group for the
state before the first event, then one delta group per state event. It
runs in one native transaction, allocating from `state_group_id_seq` as
Python does. A `prev_group` is locked against deletion and its pending
deletion pushed back, and a missing one fails with Python's message.
When the state groups share the main database and that uses the native
driver, `store_state_deltas_for_batched` calls it instead of running a
Python transaction. The context bookkeeping and the cache prefill are
shared by both paths. Room creation, and anything else sending a batch,
saves one Python transaction.
A twin test stores the same batches through both writers and compares
the rows and the groups the contexts get.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Validating a power levels event against its JSON schema took 250 us
even with the validator compiled once, ten times the rest of
`validate_new`.
Rust now answers "certainly valid" or "not sure" for the power levels
schema, the mentions schema and the canonical JSON check. Python runs
its own check only when Rust is not sure, so Python still decides every
rejection and its error message. Rust only accepts integers it parsed as
`i64` in range, so floats such as `50.0`, which the schema accepts, are
left to Python.
`_check_size_limits` takes the canonical JSON length from Rust instead
of building the dict and encoding it. Rust keeps each number's source
text, which can differ from how Python prints a float, so an event with
any number other than a plain integer is still measured in Python.
`validate_new` on a default power levels event: 305 us before, 43 us
after.
Tests generate random contents and check that whatever Rust accepts,
Python accepts too. They found that the strict canonical serializer in
`canonical_json.rs` does not range-check numbers inside arrays, so the
canonical check walks the JSON itself. Nothing else uses strict mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`create_local_event_from_event_dict` encoded the event as canonical JSON
three times: once for the content hash, once to sign the redacted event,
and once to parse it as an `Event`, which then hashed it again for the
event ID.
A Rust `LocalEventSigner`, one per homeserver, now takes the canonical
JSON once, adds the content hash and the signature, and builds the
`Event`. The signed bytes are the redacted event without its signatures,
which is also what the reference hash covers, so the event ID comes from
them without another pass. Hashes and signatures already on the event
(a make_join template) are kept, as before.
Numbers keep the text Python's encoder gave them, so floats and large
integers in old room versions hash and sign as they did. An event the
room version cannot parse is still a 400. A make_join template with a
non-object `hashes` or `signatures` entry, or no `content`, is now a
400 too, where it used to raise a 500.
Saves about 45 us per event on this host. ed25519 signing is 25 us of
what is left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
With `require_membership_for_aliases` (the default), `create_event`
checks that the sender of an `m.room.aliases` event has joined, by
loading their membership event from the database. Since the initial
events of a room are persisted in one batch, the creator's join is not
there yet, so a `/createRoom` with `m.room.aliases` in its initial
state failed with 403 "You must be in the room to create an alias for
it".
Persist the create event and the join on their own first in that case,
as for v1 rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`/createRoom` sent each invite with `update_membership_locked` after the
room's initial events, one persist round per invitee.
Invites to local users now go into the batch with the rest of the room, in
`invite` order, until the first invite that has to be sent on its own:
- an invite to a remote user, which needs a federation request first;
- an invite that would be refused (server notices user, non-admin invites
blocked, the invitee's invite rules). It is sent the old way once the room
exists and fails the request as before.
`RoomMemberHandler.build_new_room_invite_content` makes those checks and
builds the content, with the invitee's profile where the server includes
it in invites. A repeated invitee is skipped, as `deduplicate_state_event`
dropped the repeat before.
Each batched invite is also checked against the auth rules before it joins
the batch, as one that fails (inviting yourself, or a power level override
that forbids invites) would otherwise abort the whole batch and leave the
room without events. It and the invites after it are sent on their own
instead, and fail the request once the room exists, as before. The
`trusted_private_chat` preset still gives every invitee ops in rooms
without MSC4289 creators.
Modules that check invites (`user_may_invite`) or events
(`check_event_allowed`) are given the room state from the database, so
with any of those registered all invites are sent on their own. So are
invites in v1 and v2 rooms, whose create event and join are already
persisted on their own.
`invite_room_state` (and `knock_room_state`) now take earlier events of the
same batch from the batch, as they are not in the database yet. Invitees
are passed to the notifier as extra users, as `update_membership` did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
After `batch_persist_unpersisted_contexts`, an event's `EventContext` only
had state group IDs, so every read of the state before the event went to
the state store. For a new room that was eight state-group reads per
`/createRoom`: six from `deduplicate_state_event`, one from the
create-event check in `persist_and_notify_client_events`, and one from the
persister working out the new current state.
Keep the state map the batch was built from on the `EventContext`, as a
private attribute that `get_prev_state_ids` prefers and that is not
serialized, so a context sent to another worker reads the state group as
before. A context that arrives with a state group of its own, which the
batch then replaces, drops its map. `store_state_deltas_for_batched` also prefills the state group
caches with the state after the last event of the batch, through the
helper `store_state_group` now shares.
On the trial harness (SQLite) a plain room drops from 35 to 27 database
interactions, and `/createRoom` from 17 to 10 database transactions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`/createRoom` persisted the create event on its own, then the creator's
join through `update_membership`, then the rest of the initial state as a
batch. Each round paid for the persist queue, the purge lock, forward
extremities, chain cover, a replication notify and a wake-up of the stats
and presence pollers.
Build the create event and the join against the in-memory state map like
the other initial events, and persist all of them in one batch. The join's
content comes from `RoomMemberHandler.build_creator_join_content`, which
shares the suspension and profile handling with `update_membership_locked`
through the new `_prepare_membership_content`. The MSC4291 create event,
built before the room ID is known, now also leaves its context unpersisted
for the batch.
`store_state_deltas_for_batched` accepts a batch that starts the room: it
stores the empty state before the create event as a full state group, then
the delta chain. If the first context already has a group for the state
before it (a module rewrote the event), the chain builds on that instead.
`calculate_context_info` and `UnpersistedEventContext` treat an empty state
map as known empty state. Before, an empty map tripped an assertion. Room
creation never passed one, but federation could (`compute_event_context`
with an empty remote state), and such an event is now stored with an empty
state group and left to the auth checks.
Two cases keep persisting the create event and the join on their own:
v1 and v2 rooms, whose builder reads prev and auth event hashes from the
database, and servers with a `check_event_allowed` module callback, which
is given the room state from the database and may have the event rebuilt.
On the trial harness (SQLite) a plain room drops from 68 to 35 ms and 26
to 17 database transactions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`jsonschema.validate` checks the schema against the metaschema on every
call. For power levels that cost more than validating the event: about
9 ms per `/createRoom`. Check the power levels and mentions schemas once
at import and validate with bound validator instances, reporting the same
best-match error.
`create_event` also ran `validate_new` a second time on the event that
`create_new_client_event` had just validated. Drop the repeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`tests/rest/client/bench_createroom.py` times room creation per room in
the trial harness, with optional cProfile dumps and per-room counts of
database interactions and SQL statements. It skips unless
`SYNAPSE_BENCH_CREATEROOM` is set.
`scripts-dev/createroom_bench.py` times it against a real server, reusing
`persist_bench.Server`. That server now lifts the room-creation rate limit
and can run on psycopg2 instead of the native driver.
The test template database's maintenance connection now always uses
psycopg2, as `setup_test_homeserver` does. The Rust driver's `postgres`
module has no `connect`, so it failed under `SYNAPSE_TEST_RUST_DRIVER=1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`twisted.internet.task.Clock` re-sorts its whole list of pending calls on every
`callLater` and after every call it runs. A test homeserver has dozens of
pending calls and the threadless database pool schedules a `callLater(0, ...)`
per transaction, so this showed up as ~10% of a test's time in profiles.
Insert new calls in order with `bisect.insort` and only re-sort after a call
was rescheduled with `reset` or `delay`.
Together with the previous commit, `tests.rest.client.test_rooms` goes from
44.9s to 32.7s and `tests.handlers` from 146s to 114s single-process on
SQLite; a 147-test Postgres subset from 85.9s to 74.4s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Every test homeserver runs the background updates when it starts, and with a
fresh copy of the template database there are dozens to run: 29ms per test
on SQLite and 99ms on Postgres, against a median test time of ~140ms.
Run them once per process on a throwaway homeserver and copy the result back
over the template (`backup()` for the in-memory SQLite template, `CREATE
DATABASE ... TEMPLATE` for the Postgres base database), so each test's
homeserver finds nothing left to do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Trial worker memory grew steadily over a run because `HomeServer`s outlived
their test. Each one is a ~10k-object reference cycle; GC is disabled while a
test runs and the tearDown only does a gen-0 collection, which frees the
homeserver only if nothing older still points at it. Four things did:
- `ThreadedMemoryReactorClock.__init__` installs `tls._get_default_clock =
lambda: self` into the `twisted.protocols.tls` module. That held the test
reactor, whose never-removed shutdown triggers hold handlers, and so the
homeserver, until the next test's reactor replaced it. Hold the reactor
weakly instead.
- `synapse.config.cache._CACHES` maps cache names to bound `set_cache_factor`
methods, so an old homeserver lived until a later test registered a cache
with the same name. Hold bound methods through `weakref.WeakMethod`.
- `_CacheContext._cache_context_objects` is a class-level `WeakValueDictionary`
keyed on `(cache, key)`: the key strongly holds the cache, which holds the
homeserver, for as long as any context exists. Key on `id(cache)`; the id
cannot be reused while an entry exists since the context holds the cache.
- trial keeps finished `TestCase` instances around (for example in its list of
skipped tests), and those keep `self.hs`, `self.store`, ... Drop every
attribute a test added from setUp onwards before the gen-0 collection.
The fake reactor's `callWhenRunning` and shutdown hooks are emptied once the
homeserver has shut down: they never fire in tests, and the Rust runtime holds
the reactor through a reference the garbage collector cannot see, so a reactor
that still pointed at the handlers kept the homeserver alive.
With this the homeserver dies in the existing gen-0 collection. A full
collection per test was ruled out: it costs ~100ms per test. RSS over
`tests.rest.client.test_rooms` goes from 148→337 MB to 148→176 MB, and over
`tests.handlers` from 170→498 MB to 170→278 MB.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Brings Rust sliding sync (erikj/rust-sss-queries) together with Rust
event persistence (erikj/rust-persist-validate). Both branch from
a78b02b55c.
The conflicts were both sides adding parallel entries:
- the Complement and test environment switches
- the `RustHandlers` fields and getters
- the `_BlockingRustCalls` types
These keep both. `RustHandlers` builds the persister and the sliding sync
handler in one pass over the native pool. `SyncReader::begin` keeps the
persist side's flag-before-BEGIN and the sliding sync side's transaction
count and `begin_write`. Both sides had added the `invite_room_state`
and `knock_room_state` accessors to `Unsigned`, so one copy is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
The native driver's write path recorded no `sql_txn_count` or
`sql_txn_duration`, nor the log context's transaction counts, so the
`persist_events` panels read near zero at `rust_event_persistence:
transaction` and `full`.
`run_write_interaction` now times each interaction as `new_transaction`
does: every attempt, from when it has its connection, so the wait for
one is left out. The times go into a task-local list that
`with_txn_timings` collects. The persister's entry points carry the list
back on what they already return (`PersistEffects`, `PreparedChunk`,
`PersistedBatch`, `CurrentStateUpdate`), and Python records it where
`new_transaction` would: the two counters, the pool's perf counters and
the current log context, which is the caller's when Python applies what
Rust returned. That costs no GIL hop.
When a persist fails, or Python gives up a batch, no result carries the
times back, so the persister keeps them and `flush_rust_persist_metrics`
counts them at the next scrape, without a log context. In shadow mode
Python's own interactions are the ones used, so Rust's dry run is not
recorded. The stream orderings' `nextval`, which runs outside a write
interaction, is recorded as `_load_next_mult_id`, as Python's is. The
`db.<desc>` tracing span is not recorded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`_update_current_state`, the queue's `_UpdateCurrentStateTask`, becomes
one call to `Persister.update_current_state`:
- `calculate_current_state` reads the room's forward extremities (its
state DAG extremities in an MSC4242 room) and takes the state of
their one state group, or resolves several, after checking the groups
still exist and putting off their deletion, as `resolve_state_groups`
does.
- The delta from the current state and the sliding sync table changes
are computed as for a chunk, with no events.
- If anything changed, one stream ID is taken from the events stream,
as `get_next` does, and `_update_current_state_txn` runs in Rust.
Python applies the effects, which finishes the ID.
A room on state resolution v1 whose extremities need resolving is handed
back, and Python recalculates its state as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`_after_persisting_events` moves into the batch step, after the write
and before Python applies the chunk and finishes its stream orderings,
as it ran inside `get_next_mult` in Python:
- The persisted events, by type and origin, and the persisted position
go to the Rust metrics that `flush_rust_persist_metrics` reports.
- With MSC4293, a join, or a leave by someone else, ends the redaction
of the target's events. A ban or kick with `redact_events` set, by a
sender who may redact (`_can_sender_redact`, read from the current
state), starts it. Python then drops the room's events from its event
cache, through a new `event_cache_room_invalidations` effect, even if
a later event of the chunk fails, as Python drops them right after
each start.
- The forward extremities prefill moves into the effects.
The MSC4293 flag goes with each batch, as Python reads it per chunk and
tests change it at runtime.
If what follows the write fails, the chunk is still written, so Python
applies its effects and finishes its orderings as a failure, as
`get_next_mult` does when its block raises, then raises the error and
releases the batch's state groups as a failed persist does.
Where Python would raise on odd power levels or memberships, Rust is
more lenient: it reads a level that is a string of an integer as that
number (Python compares two strings, or raises), a null default as 0
and a `users` or `events` that is not an object as missing, and skips a
member event without a membership. Numbers, floats and bools compare as
in Python.
The chunk path, for chunks Rust hands back, keeps the Python block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
At `full`, `_persist_event_batch` hands the batch to
`Persister.persist_batch` and steps through it with `batch_next`:
- The first step drops the events already persisted under the same
transaction ID (`get_already_persisted_events`).
- Each step prepares the next chunk of up to 100 events, marks the
batch's state groups as being persisted
(`persisting_state_group_references`, two short native
transactions), takes the chunk's stream orderings and writes it.
- Python applies what the chunk asks for, runs
`_after_persisting_events` and finishes the orderings before the next
step, which first releases the state groups. So each chunk is
prepared against current caches.
- If Python cannot apply a chunk, `batch_abort` releases the state
groups as a failed persist does.
A chunk Rust cannot prepare (a state resolution fallback, or an error)
is handed back with the rest of the batch to the chunk-by-chunk path,
which prepares it again and falls back or raises as at lower levels.
Python drives the steps, rather than Rust calling back into Python, so
every step has a `_blocking` twin for tests and nothing waits on the
reactor from Rust. That costs the same hops as a callback: one in and
one out per chunk.
The prepare inputs are now built from `Event`s that share their data
with the Python events, so a chunk's input can be built without the
GIL once the batch has been deduplicated.
The persister's Deferreds already follow the logcontext rules, so they
are awaited directly. Wrapping them in `make_deferred_yieldable` again,
as the `prepare_chunk`, `persist_events_txn` and `persist_chunk` call
sites did, resumed the caller in the sentinel context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
At `full`, `Persister.persist_chunk` takes the chunk's stream orderings
from the events or backfill stream tracker, which Python's
`_stream_id_gen` and `_backfill_id_gen` share. It runs `nextval` in
autocommit after prepare, as late as `get_next_mult` does, and sets each
event's stream ordering, received time and instance before the write
transaction. Backfilled events take theirs negated.
The orderings come back unfinished in `PersistEffects.stream_ids`.
`_persist_events_and_state_updates` skips `get_next_mult` when Rust takes
them. Its post-transaction block (metrics, MSC4293
`room_ban_redactions`, the extremities prefill) moves to
`_after_persisting_events`, which both paths call. At `full` it runs
after the effects are applied and before the orderings are finished, as
it ran inside `get_next_mult`. So a ban that redacts a user's events is
recorded before other workers see it.
`finish_rust_persist_stream_ids` finishes the orderings as
`get_next_mult` does on exit: it tells replication, and on success moves
`stream_positions` on. If the transaction fails, Rust finishes them. If
the apply or the follow-ups raise, Python finishes them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The seventh commit of phase 2, which completes it. The Rust transaction
now covers the whole of `_persist_events_txn`, so at the `transaction`
level every chunk Rust prepared is written in Rust, including on a
server with `include_profile_updates_in_sync`.
- `txn/profile.rs` ports `_record_profile_updates_for_state_delta_txn`
and the helpers it calls: the joined room updates of local members
who joined, and the left room updates, both ways, of members who left
or were state reset out of the room, with the deletes of the rows they
replace. It runs at the end of `update_current_state`, as in Python.
- Its rows take IDs from the profile updates stream inside the
transaction, through a third `TxnStream` over the tracker of
`_profile_updates_id_gen`. `PersistEffects.profile_stream_ids` hands
them to the applier, which finishes them as `get_next_mult_txn` does,
as it does the sticky events stream's. The two share
`_finish_txn_stream_ids`.
- The controller no longer falls back to Python on a server that
records profile updates.
- The Python helpers take the members in user ID order where they took
a set's order, so both sides take the stream IDs in the same order.
- `Persister.persist_events_txn` resolves with the effects alone. The
events left and the redacted copies were only for the Python steps
after Rust's in the twin test, which are gone.
The twin test compares the profile updates rows each twin's persists
wrote, and the rows of each twin's own users from outside them, which a
leave deletes. Those are compared without `inserted_ts`, as each twin
writes them at a different time. New scenarios, with members of each
twin's own, cover:
- joins, a change of display name, and leaves
- several members in one chunk
- a member state reset out of the room
- a remote member state reset out of it, which writes nothing, as only
local members have membership snapshots
- a leave while still sharing another room with some members, and with
all of them
- a leave deleting the rows about earlier profile changes
- the last local member leaving
Making the Rust shared room query match nothing, dropping the reverse
left room rows, or skipping the delete of the leaver's rows about the
others, fails the scenarios that cover it. The
prereqs test now checks that a server recording profile updates writes
the rows in Rust and finishes the stream IDs.
These suites pass at `SYNAPSE_TEST_RUST_PERSIST=transaction` on the
native driver: `tests.handlers` (including the profile and sync tests
that record profile updates), `tests.federation`, `tests.replication`,
`tests.rest.client.sliding_sync` (including the profiles extension),
the phase 2 storage suites, and the `tests.rest.client` group but for
the two `test_rooms` tests that count database transactions, which fail
the same way at `prepare`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The sixth commit of phase 2. At the `transaction` level, a chunk Rust
prepared is written by the Rust transaction in place of
`_persist_events_txn`.
- `Persister.persist_events_txn`, with a `_blocking` twin for tests,
runs the transaction over the chunk and resolves with its effects.
The test-only entry it replaces shared its body.
- `_persist_events_and_state_updates` takes a `RustTransaction` (the
persister and the prepared chunk). With one, it calls Rust inside the
same stream ordering allocation, then applies the effects, so the
event stream IDs are finished after them, as before. The call and
the apply run together under `delay_cancellation`, as
`runInteraction` runs its callbacks: a transaction that commits must
have its effects applied, or the cache and sticky events stream IDs it
took are never finished.
- The controller passes one when the level is `transaction` or later
and Rust prepared the chunk. A chunk Python prepared, after a state
resolution fallback, is written in Python. So is every chunk on a
server with `include_profile_updates_in_sync`, as the Rust
transaction leaves the profile updates to Python.
- `_add_to_cache` moves to Rust: the transaction picks the events
neither rejected nor redacted, in
`PersistEffects.event_cache_prefills`, and the applier puts the events
Python holds into the event cache. Of a duplicated event, it puts the
copy that was written, the earliest non-outlier, as `_add_to_cache`
reads the deduplicated chunk.
- The applier is now `apply_rust_persist_effects(store, ...)` in
`synapse/storage/databases/main/rust_persist.py`, as the event
persister may be a worker, whose main store is a `GenericWorkerStore`.
A mixin on both stores cannot be linearised into `DataStore`. If a
step raises, it still finishes the cache and sticky events stream IDs
it has not finished, as `call_on_exception` does in Python.
The twin test compares the events each side puts in the event cache,
and checks which copy of a duplicated event goes in. New prereqs tests
check:
- that Rust writes the chunk, with Python's transaction made to fail,
and the applier fills the event cache
- that a lowered level, or a server that records profile updates,
writes in Python a chunk Rust prepared
- that the transaction takes the GIL at no persist site
- that cancelling the persist mid-transaction waits for it and applies
the effects
- that a worker event persister writes in Rust to the main database
The Rust transaction does not yet record the `sql_txn_*` metrics, the
log context's transaction counts or the `db.persist_events` span, as
the native driver's write path records none of them.
These suites pass at `SYNAPSE_TEST_RUST_PERSIST=transaction` on the
native driver: `tests.handlers`, `tests.federation`, `tests.replication`
(sharded event persisters included), `tests.rest.client.sliding_sync`,
the storage suites in the phase 2 plan, and `test_sync`,
`test_relations`, `test_redactions`, `test_sticky_events`,
`test_profile`, `test_retention` and `test_rooms` in
`tests.rest.client`, but for two `test_rooms` tests that count database
transactions. Those fail the same way at `prepare`: Rust's prepare step
takes one more transaction per room creation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
A context sent to the event persister over replication holds its state
maps as `immutabledict`s, which Rust's persist context conversion
refused, as it took only a `dict`. So on a deployment with workers,
preparing a chunk in Rust failed for every event another worker sent,
and fell back to Python with an error logged. It now takes any mapping.
Two worker tests with the native driver showed it, and then showed a
test harness problem. `make_worker_hs` points a worker's Python pool at
the master's database once the worker is built, but the worker's Rust
handlers keep the native pool they were built with, on the worker's own
database. It now points that pool at the master's database too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The fifth commit of phase 2. The Rust transaction now runs the rest of
`_persist_events_txn`, but for the event cache prefill (`_add_to_cache`)
and the profile updates, which stay in Python.
- `txn/sticky.rs`: `insert_sticky_events_txn` for the chunk's new
events, and `un_soft_fail_sticky_events_txn` for the events
`compute_sticky_events_to_un_soft_fail` found in prepare. A
de-outliered sticky event becomes sticky inside `update_outliers`, as
in Python, so the chunk's redactions come after it. As in Python, an
event with a zero sticky duration is not tried there.
- `txn/sliding_sync.rs`: the latest stream ordering and bump stamp of
the room's `sliding_sync_joined_rooms` row.
- `txn/streams.rs`: what was `CacheStream`, now `TxnStream` for any
stream the transaction takes IDs from, several at a time. The sticky
events stream is the second. Its IDs come back unfinished in
`PersistEffects.sticky_stream_ids`. Python finishes them once the
caches are invalidated, notifies replication and, as the stream has
writers, updates `stream_positions`, as `get_next_mult_txn`'s
callbacks do. A failed attempt's IDs are finished as for the cache
stream.
`persist_events_txn` returns the events left. The test entry no longer
returns the de-outliered events.
Python's `un_soft_fail_sticky_events_txn` gave the events their new
stream IDs in a set's order. It now keeps their order in the stream, by
`event_stream_ordering`, with any event without a row last, by event ID.
Rust does the same, which lets the twin test compare several events
lifted at once.
The new scenarios found a Python bug, which Rust keeps. A de-outliered
sticky event's `sticky_events` row has the stream ordering of the chunk
that de-outliered it, but its `events` row keeps the one it was first
persisted with. `delete_sticky_events_txn` looks for the latter, so a
redaction of the event, in that chunk or any later one, leaves it
sticky until it expires.
The twin harness:
- records the stream IDs the applier finishes and the notifiers it
calls, rather than modelling them from the effects
- merges each persist's finished IDs per stream into one call, and
compares its notifiers by whether they were called. Python queues
them per allocation of IDs, and the effects do each once.
- names a background process by what it runs: an ID generator's
`stream_positions` update by its stream, any other by its description
New twin scenarios:
- sticky events in one chunk, one of them expired
- a sticky outlier, de-outliered; de-outliered and redacted in one
chunk; de-outliered with a new sticky event in one chunk
- soft-failed sticky events lifted when their sender joins, one and
three at once
- a rejected sticky event, a backfilled one, and one redacted in its
own chunk
A prereqs test checks that applying the effects finishes the sticky
events stream IDs after the event cache invalidations, then notifies,
and updates `stream_positions`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The fourth commit of phase 2. The Rust transaction now also runs
`_update_current_state_txn`, apart from the profile updates at its end.
- `txn/current_state.rs`: the delta stream rows, read before the current
state changes, then `current_state_events` and the state columns of
`sliding_sync_joined_rooms`. When the server leaves the room, the room
version is recorded and the room's current state goes instead.
`local_current_membership` and the membership snapshots follow either
way. The effects move the current state delta and membership stream
change caches on, and invalidate the state caches. The `cs_cache_fake`
rows name 50 members each. Remote members who share no room with the
server any more have their device lists unsubscribed.
- `txn/caches.rs`: `_send_invalidation_to_replication`, the cache stream
row without the local invalidation.
The profile updates take IDs from the `profile_updates` stream, which
has writers and so a `stream_positions` row. They move to
`_record_profile_updates_for_state_delta_txn` in Python, which the twin
test runs after Rust, and are left for a later commit. The twin test
cannot compare their rows, as the twins share the profile tables. The
profile suites cover the moved code.
Rust differs from Python in these places:
- Rust writes the members of the `cs_cache_fake` rows, and the device
list rows of the remote users who left, in sorted order. Python's
follow a set's.
- A create event whose `room_version` is not a string, null included,
or whose `creator` is neither a string nor null, fails the persist
when the server leaves. Python stores a number cast to text or a
NULL, and fails on an object. Event auth requires a string
`room_version`, so only old or corrupt data can differ.
The room version upsert has a Postgres test on temporary tables. The
twin test cannot see it, as the `rooms` row always holds the version
already.
The twin harness now applies Rust's effects with
`_apply_rust_persist_effects`, as the facade will, so the steps after
Rust's see the caches Rust invalidated. Some of what Python writes from
a set, it compares in any order:
- sets in effects, as sorted tuples
- a run of `cs_cache_fake` rows, as its members and its row sizes
- a run of single-key cache stream rows of one cache, as its keys
- an event's `auth_events`, where the stored JSON matches its content
hash, which covers their order. The hand-built events are rehashed.
It compares `device_lists_remote_extremeties` on each twin's own remote
users.
New twin scenarios:
- the sliding sync state columns (encryption, name, tombstone and a
leave)
- the server leaving the room, with one or two remote members still
joined
- remote members and their device lists
- 60 members changing at once
- a state key, and a local member's invite with its snapshot, removed
by state resolution
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The third commit of phase 2. The Rust transaction now also runs
`_update_metadata_tables_txn`, apart from the event cache prefill at its
end (`_add_to_cache`), which stays in Python:
- `txn/metadata.rs`: push actions move from the staging table, and the
chunk's staging rows go. A redaction removes the push actions and the
relation of the event it redacts, and moves or deletes its thread.
Then search (with the plain text topic from `m.topic`), redactions,
retention, relations and threads, labels, expiry, and room members.
An out-of-band membership of a local user also updates
`local_current_membership` and the user's sliding sync snapshot, with
the stripped state of an invite or knock.
- `txn/edges.rs`: `event_edges` with the backward extremities, and the
MSC4242 state DAG edges.
- `txn/caches.rs`: `_invalidate_cache_and_stream`. The rows go into
`cache_invalidation_stream_by_instance` inside the transaction, with
IDs from the cache stream's tracker. The committed attempt hands its
IDs to Python in `PersistEffects.cache_stream_ids`. Python finishes
them after invalidating the caches, as `get_next_txn` does. A failed
attempt's IDs are finished when its steps fail, or for a failed
commit, by the next attempt or when the persist ends. A fetch of an ID
that is dropped before the sequence answers still ends, so a cancelled
persist cannot hold the stream's position back.
Where Python and Rust differ, the input is malformed:
- Python stores labels from any value it can iterate, and non-string
labels. Rust stores only string labels from a list.
- An integer lifetime or expiry beyond `u64` is a float to serde, so
Rust ignores it where Python's insert fails. Within `u64` but beyond
`bigint`, both fail.
The persister reads `enable_search` and `enable_ephemeral_messages`
from the server config. `persist_events_txn_for_test` takes
`inhibit_local_membership_updates`.
New twin scenarios: each kind of relation and its redaction, which
moves the thread back and then deletes it, a relation redacted in its
own chunk, two thread replies in one chunk, search, retention (also two
policies in one chunk), labels and expiry, member rows, push actions and
their redaction, an out-of-band invite and its rejection, a knock, and a
backfilled invite. The out-of-band ones are skipped in state DAG rooms,
as a hand-built event has no state DAG context. A redaction before its
target is now stored with a recheck still needed.
The test harness compared `event_edges` by its `room_id` column, which
is no longer filled in, so it compared no rows. It now compares by
`event_id`. It also names the cache behind each invalidation, compares
the cache stream rows' times, and finishes the cache stream IDs Rust
hands back. Fifteen mutants of the new steps each fail the comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The second commit of phase 2. The Rust transaction now runs the steps of
`_persist_events_txn` from `_update_room_depths_txn` up to
`_store_rejected_events_txn`:
- `txn/events.rs`: the room's depth and its place in the events stream
cache, de-outliering, redactions that already apply (the event is
stored pruned) and the earlier events the chunk's redactions confirm
(they stop being sticky, and a passed recheck is stored), the
`event_json`, `events` and `state_events` rows, transaction IDs, state
groups and partial state, the chain cover index and `event_auth`, and
rejections.
- `txn/edges.rs`: the forward and state DAG extremities, and the
backward extremities and timeline gaps of a de-outliered event.
`_handle_mult_prev_events` will use the latter for new events.
- `txn/json.rs`: JSON as `json_encoder` writes it, with non-ASCII and
DEL escaped. The event and its internal metadata keep the key order
`get_dict_for_persistence()` and `get_dict()` give, so the `event_json`
text is Python's byte for byte. Only number text from federation can
differ (`1.50` against Python's `1.5`).
An event with partial state in a room that is no longer partial stated
fails with `PartialStateConflictError`, as in Python.
`Persister.persist_events_txn_for_test` takes the chunk Rust prepared,
which `PreparedChunk` now shares through an `Arc`. It returns which
events were stored redacted and which were de-outliered, so the twin
test's Python half can carry on from there. Making a de-outliered event
sticky stays in Python until the sticky commit, as it needs the sticky
stream's ID generator. Python makes those sticky before deleting
redacted events' sticky rows, so the sticky commit has to insert them
inside `update_outliers`.
New twin scenarios: transaction IDs, URLs and non-ASCII text, a
redaction recheck (alone and in a chunk with its target), a partial
state conflict, a redacted sticky event, and a redaction of a later
redaction, which needs censoring again, and an event de-outliered while
its prev event is an outlier (a backward extremity and a timeline gap).
The test checks that room B's transaction takes the chunk prepared for
it. The fork scenario now advances
the clock between its topics, as state resolution otherwise breaks the
tie by event ID, which differs between the rooms. Each ID generator's
stream is ranked separately. Nineteen mutants of the new steps each
fail the comparison. `PersistenceJsonTestCase` compares the `event_json`
text byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The first commit of phase 2 of the event persistence plan
(`transaction` level). `rust/src/persist/txn/` will hold the port of
`_persist_events_txn`. This commit adds its driver: the share lock on
the room's `rooms` row, the cache updates Python queues for every event
up front, and dropping duplicate events. It returns the events left for
the steps still in Python.
`tests/storage/test_rust_persist_txn.py` builds the same DAG in two
rooms. Room A persists in Python. Room B prepares in Rust, runs the
Rust part of the transaction in its own committed transaction through
`Persister.persist_events_txn_for_test`, and then runs the Python steps
after it. The test compares the rows of every table the transaction
writes, after normalising room and event IDs, stream positions, state
groups and chain IDs, and compares the cache updates both rooms asked
for. Each later commit moves more steps from the Python half to the
Rust half, in transaction order, until the facade runs the whole
transaction. Mutating the driver (dropping the redaction
invalidation, the relation, the state key, or the outlier rule for
duplicates) fails the comparison.
Two open items from phase 1:
- Reads before the transaction now retry. Python runs each read in a
`runInteraction` that retries a lost connection or a cancelled
statement. `Persister.prepare_chunk` now prepares the whole chunk
again on such an error, up to five attempts. That is safe because its
only write, bumping the state groups it uses, can be repeated. The
reader and the state resolution store keep the pool's error as the
cause, so `is_retryable_error` can classify it. A `WriteError` does
not count, as its write interaction has already retried, and
`WriteError::from_pg` shares the classification.
- The Rust persister needs the state tables in the main database, as
it reads and writes both in one transaction. `Persister::for_homeserver`
now checks for that, and the persister is off when they are apart.
Also fixed in the dispatch: at `off`, `_prepare_chunk` ran Rust when a
persister existed. That only happens when a test changes the level at
runtime. The twin test does exactly that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`Persister.prepare_chunk` runs the steps of the previous commits for a
chunk: the extremities and state delta, then the sliding sync table
changes and sticky events, then the chain cover index. The chain cover
step comes last as the only one that writes, so a failure in an earlier
step leaves nothing for Python to find. Sliding sync and sticky events
now come before the stream orderings are allocated, as nothing there
reads them. Each step reads the clock when it needs the time.
`_persist_event_batch` dispatches on `rust_event_persistence`:
- `shadow`: Rust runs first with its chain cover write rolled back, then
Python. The results are compared and Python's are used. The comparison
ignores orders Python leaves to chance and the IDs of new chains. Both
rejecting the chunk with the same error is a match.
- `prepare` and above: Rust's results are used. A `SynapseError` is
raised as Python would raise it. Python computes the chunk when Rust
cannot resolve state in the room, or on any other Rust error.
Trial runs strict by default (`SYNAPSE_TEST_RUST_PERSIST_STRICT`), so a
mismatch or Rust error fails the test. Rust's counts and state res stats
are recorded by Python once it uses the result.
At `prepare`, the `_calculate_state_and_extrem` and
`calculate_chain_cover_index_for_events` timings give way to
`rust_prepare_chunk`. Python's state resolution and state group caches
are not filled, so creating a room costs one more transaction
(`RoomsCreateTestCase` counts it) and the first local event after a
fork resolves state again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
Port `compute_sticky_events_to_un_soft_fail` and
`_get_soft_failed_sticky_events_to_recheck` to
`rust/src/persist/sticky.rs`. The reads go through a `StickyReads`
trait over the persist reader. Events to recheck and auth events not in
the chunk come from the Rust event loader, redacted and without
rejected events, as `get_events` returns them.
An auth failure keeps an event soft failed. A malformed event fails the
step, as Python raises it as something other than `AuthError`.
`persist_error_to_py` now raises an `AuthFailure` as the auth rules
raise it.
`StickyTestCase` compares the two over the same database, chunk and
delta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
The phase 6 bench found Rust making 2 to 6 times Python's queries: 8 per
room computed, and 4 on an incremental request where nothing changed.
- Bundled aggregations read the thread summary, edits and references
through Rust caches that mirror Python's `get_thread_summary`,
`get_applicable_edit` and `get_references_for_event`, so Python's
invalidations reach them. The thread summary takes one statement with a
window count, where it took a transaction of up to two. v2 sync in Rust
shares these reads.
- `ignored_users` reads through a mirror of Python's cache.
- The `instance_map` table is cached. It only gains rows, so a stored
token naming a writer the cache lacks drops the cache and reads again.
- The self-leave query is skipped when the membership stream cache says
the user's membership has not changed since the `to_token`, the
shortcut Python's query carries as a TODO.
- On a room's first send, the name and the required state come from one
state read. Rooms with heroes keep a second read for their memberships.
On the bench, initial sync goes from 173 queries to 49, the range
expansion from 662 to 178 (Python makes 335), and a no-op incremental
from 4 to 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KETBASkkVomGNnbut9NWQq
Rust's `encode_rooms` returned a dict per room, which Python then
JSON-encoded again. On the bench's range expansion those two steps were
17 of Rust's 31 ms under the GIL.
`encode_rooms` now returns the rooms as one `bytes` value of JSON object
members and encodes them with the GIL released. The servlet encodes the
rest of the response on a thread, as `respond_with_json` did, splices
Rust's rooms into the rooms object next to any Python fallback rooms, and
sends the body with `respond_with_json_bytes`.
The bench counts the encoder's time as off the GIL. Rust's reactor time
on the range expansion goes from 0.13 of Python's to 0.06.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
- Count Rust's calls back into Python (the events stream cache) as reactor
time: they hold the GIL in production too.
- Profile only the timed request, not the setup request or the message
sent before it.
- Measure what the plan meant by "500 rooms sent": a [0,19] request on a
connection that sent 500 rooms before. The [0,499] list stays as its own
scenario.
- Say that trial runs Python's database functions on the reactor thread,
so reactor CPU is the GIL time of both sides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
Port `_calculate_sliding_sync_table_changes` and
`_get_sliding_sync_insert_values_from_state_map` to
`rust/src/persist/sliding_sync.rs`. The reads go through a
`SlidingSyncReads` trait over the persist reader. State events that are
not in the chunk come from the Rust event loader, redacted and without
rejected events, as `get_events` returns them.
The membership snapshots to insert are sorted by user ID. Python lists
them in the order of the delta.
The stripped state helper only runs inside the write transaction, for
out-of-band memberships, so it moves to phase 2.
`SlidingSyncTestCase` compares the two over the same database, chunk
and delta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
An opt-in trial module that times each request on a Python and a Rust
connection of one user, in turn, for the scenarios of phase 6 of the
sliding sync plan: initial sync, incremental with nothing or one event
changed, a range expansion, a 2000-room account and a connection with 500
rooms sent, each with and without the account data and receipts
extensions. It reports wall time, reactor-thread CPU less the time inside
Rust calls, process CPU, and database time, queries and transactions, and
can dump a cProfile per scenario.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
Port `calculate_chain_cover_index_for_events` and the class methods it
calls from `PersistEventsStore` to `rust/src/persist/chain_cover.rs`,
with `_LinkMap` and `sorted_topologically`. As in Python, the step runs
in a write transaction of its own. It allocates chain IDs from the
`event_auth_chain_id` sequence and puts off events whose auth events are
missing, but does not write the index.
The reads and writes go through a `ChainCoverStore` trait, which
`WriteTxn` implements and the unit tests implement in memory. The step
takes a `ChainCoverEvent` view that reads the auth events up front but
fails only when they are used. An MSC4242 event can't give its auth
events before auth has run, and Python reads them only for events not
yet in the index.
The result matches Python's except in two orders Python leaves to
chance: which new chain gets which ID, and the order of an event's
links. The redundant-link reduction stays quadratic, as in Python.
`PersistReaderForTest` gains the step, and `ChainCoverTestCase` compares
it with Python on the same database. New chains are compared by the
lowest event ID on them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
- Re-raise SlidingSyncUnknownPosition from finish_request. A racing
request on the same connection can clear the position before the
persist, and Python would fail the same way, so a Python retry only
repeats the work.
- Treat an empty events stream cache as a miss, as Python's
get_entities_changed does, so Rust asks the database.
- Copy the room statuses out of the connection state before building
Python objects, so the lock is never held across a Python call.
- Trace the rooms Python computes for Rust, as the Python path does.
- Test the per-room fallback: Python computes a room Rust leaves to it,
and its changes reach the Rust state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
Port `_get_new_state_after_events`, `_calculate_state_delta`,
`_is_server_still_joined` and the two steps that call them with the
extremity calculations, `_calculate_new_forward_extremities_and_state_delta`
and `_process_state_dag_forward_extremities_and_state_delta`, to
`rust/src/persist/state_delta.rs`. Reads go through a `StateDeltaReads`
trait on top of `ExtremityReads`, and state resolution through a
`StateResolver`, so the unit tests can fake both.
`NativeStateResolver` checks and bumps the input state groups as
`check_state_groups_and_bump_deletion` does, then resolves with the Rust
state resolution on the native pool, through `resolve_on_pool`, which is
split out of `StateResolutionHandler::start_converted`. The persister
shares the handler's conflict cache. There is no `_state_cache` of whole
resolutions. Rooms on state resolution v1 or without a chain cover index
fail with `StateResFallback`, for the caller to run the Python step.
`NewEvent` gains the fields the state delta reads from an event and its
context. `PersistReaderForTest` gains the five calculations, and
`test_persist_reader.py` compares each with the Python method on events
built by the event creation handler, including a resolved fork and an
MSC4242 room.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
In `rust_sliding_sync: on` mode, `current_sync_for_user` is now four Rust
calls around the Python extensions: load the connection state, compute
the interested rooms (the port of `_compute_interested_rooms_new_tables`),
compute the rooms, and finish the request (unsent and sent rooms, then
the persist).
The connection state is a Rust pyclass. The extensions read and update it
through views with the surface of Python's `PerConnectionState` and
`MutablePerConnectionState`. Receipts tokens stay in their database form
in Rust and become `MultiWriterStreamToken`s only when Python asks.
The whole request falls back to Python when the sliding sync tables are
not filled, for a filter Python does not implement either, and on any
Rust error. Rooms Rust leaves to Python run `get_room_sync_data` over a
Python copy of their slice of the state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
Port `_calculate_new_extremities`, `_calculate_new_state_dag_extremities`
and `_prune_extremities` from the persist events controller to
`rust/src/persist/extremities.rs`. They read through an `ExtremityReads`
trait, which the `PersistReader` implements and the unit tests implement
over an in-memory DAG. Metrics go to a `PersistCounts` the caller owns,
so shadow mode can drop them.
Missing prev state events fail with `MissingPrevStateEvents`, which
`persist_error_to_py` turns into the same 500 `SynapseError` as Python.
`PersistReaderForTest` gains the three calculations, and
`test_persist_reader.py` compares each with the Python method on the
same database and the same new events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
The differential test now also checks that the Python and Rust sides
make the same changes to their connection's state: the room configs and
the lazy-loaded members. It also covers an avatar change, a ban, a
rejected remote invite, a state reset out of a room, and a prev_batch
token that names another writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`PersistReader` wraps a `SyncReader` and adds the reads the persist
controller makes before its write transaction: forward and state DAG
extremities, the prev event walks of `_get_events_which_are_prevs` and
`_get_prevs_before_rejected`, `have_seen_events`, memberships by event ID,
`is_local_host_in_room_ignoring_users`, the room version, the sliding sync
joined room check, `get_last_event_pos_in_room` and the soft failed sticky
events. None of them is cached. Each runs once per chunk.
The chain cover reads are free functions over a `Querier`, not reader
methods. The chain cover step reads, allocates chain IDs and writes in one
transaction, so they must run on that transaction's connection. To allow
that, `WriteTxn` now implements `Querier`. A `WriteError` that passes
through `anyhow` on the way out comes back as itself, so a conflict is still
retried.
The persister builds its own `SyncStore` over the shared caches.
`tests/storage/test_persist_reader.py` runs each read against the Python
method or statement it ports, on the native driver.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
`max_event_pos` now takes `get_max_pos_of_last_change` for the rooms the
events stream cache knows, as `_bulk_get_max_event_pos` does. The port
keeps Python's quirk: the cache answers only when it knows every missing
room, because Python's batch loop goes over all of them.
The membership change tests now add delta rows without an event, as the
server leaving a room or a state reset write them. The branches that
handle those rows were not reached before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
Phase 0 of the event persistence plan, apart from the write transaction
helper:
- `experimental_features.rust_event_persistence`, one ordered level
(off, shadow, prepare, transaction, full). Nothing uses the levels yet
beyond building the persister.
- `RustHandlers.persister`: built on the events writer when the level is not
off and the main database uses the native driver. It holds the pool, the
shared caches and the events, backfill and cache stream trackers.
- `PersistEffects`, the Python cache updates a Rust transaction will ask for
in place of `txn.call_after`, applied in one call by
`DataStore._apply_rust_persist_effects`. A test persists batches through
Python and checks that applying the matching effects updates the same
caches. The chunk's stream IDs come back unfinished and are finished last,
as Python's `get_next_mult` does after the transaction's callbacks, so
the current token never passes an event the stream change caches don't
know about.
- `PersistContext`, read from an `EventContext` once per event.
- `Event::to_persistence_value` and `EventInternalMetadata::to_persistence_value`:
the `event_json` columns built without the GIL.
- Rust-side persistence counters, added to the Python metrics by a pre-collect
hook. A homeserver can now register more than one such hook. The persisted
position is reported only when it moved, so it can't pull back the gauge
the Python path sets.
- `SYNAPSE_TEST_RUST_PERSIST` for trial and `RUST_PERSIST` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
`MultiWriterIdGenerator` tracks in memory which IDs this instance has
allocated and not finished, and each writer's position, and derives the
current token from them. That state now lives in a Rust `StreamIdTracker`,
so that Rust code can allocate and finish stream IDs without the GIL. The
generator keeps what touches the database or the reactor: loading the start
positions, the sequence, the `stream_positions` table and notifying
replication.
The logic is a line-by-line port. Two changes in form: the in-flight fetches
are a counted map (they were a sorted multiset), and the position gauge reads
the tracker when scraped instead of being set on each change. The issue 19795
debug line is logged after the lock is released, since logging can take the
GIL.
One behaviour change, a fix: on restart, the loader sorted the rows past
`stream_positions` by their stored ID before negating them. On the backfill
stream that visited our own rows furthest first, so our position ended at
the nearest of them. The rows are now negated first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
`SlidingSyncReader` gains the reads and writes of a connection's
per-request state:
- `connection_lazy_members`: `get_sliding_sync_connection_lazy_members`
- `load_connection_state`: the `get_and_clear_connection_positions`
transaction. It bumps `last_used_ts`, deletes the other positions,
un-forks the lazy members, drops the unused required state rows and
reads the state.
- `persist_connection_state`: the `persist_per_connection_state`
transaction, from the row lock through the lazy member upserts, with
`UNNEST` for the batched upserts and deletes
The state crosses as `StoredConnectionState` and `ConnectionStateUpdates`,
with each stream's tokens as strings, as `PerConnectionStateDB` holds
them. Parsing the tokens is the caller's job once the state is Rust-owned.
An unknown position is the typed `UnknownPosition`, which the test entry
points raise as `SlidingSyncUnknownPosition`.
Both run as `REPEATABLE READ` read-write transactions, retried up to
five times on a serialization failure or a deadlock, as `runInteraction`
does. The caller passes the time in, so the tests drive both sides from
one clock. Loaded states are memoized by their arguments in a cache
named after Python's `@cached` one, which nothing invalidates. A
long-poll's second load and a client's retry therefore run the
transaction once.
Rooms share required state rows by their JSON. `encode_required_state`
writes it as Python's `json_encoder` does, ASCII-escaped, so that rows
written by either side dedup.
The test runs twelve random operations through Python on one connection
and through Rust on another, 20 seeds. The operations are persists on
the latest position, on a fork or from scratch, and loads of current,
stale and foreign positions. After each step it compares the two
connections' rows (positions by order, required state IDs by their
JSON) and what each load returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/storage/sliding_sync_store.rs` adds `SlidingSyncReader`. It is
a `SyncReader` with the sliding sync queries on top, so one request
counts its v2 and sliding sync reads in one set of stats. Only the path
Python takes once the background updates have finished is ported.
- membership snapshots: `rooms_from_snapshots`, `self_leave_rooms_after`,
`snapshot_room` and `snapshot_rooms`
- membership changes: `snapshot_membership_changes` and
`membership_delta_changes`. The caller passes the membership stream
cache's answer in as a bool.
- stream positions: `rooms_with_updates_since`, which takes the events
stream cache's changed rooms when it has them, and `max_event_pos` and
`last_event_pos_before`
- room metadata: `room_type`, `room_version_id`, `partial_rooms`,
`tags_for_user`, `cached_member_counts` and `is_encrypted_at`
Each Python `@cached` method behind these gets a Rust cache that mirrors
it by name: the snapshots, `_get_max_event_pos`, `get_member_counts`,
`get_room_type`, `get_room_version_id`, `get_partial_rooms`,
`get_tags_for_user` and `get_global_account_data_by_type_for_user`. The
v2 reader's global account data read now goes through that last one.
`get_partial_rooms` takes no arguments, so `()` becomes a cache key.
A row's null instance name reads as "master", as everywhere else in
Rust. Python keeps the `None` in `get_sliding_sync_membership_changes`.
That only matters for a token whose instance map names "master".
`tests/storage/test_sliding_sync_store.py` runs each query against the
Python method on the same database, with the writers scattered over
four instances for the token-bounded reads.
Python's `get_rooms_that_have_updates_since_sliding_sync_table` calls
`have_finished_sliding_sync_background_jobs()` without awaiting it. The
coroutine is always truthy, so Python never takes its "assume all rooms
changed" branch. The Rust port does not have that branch either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/extensions/relevant.rs` holds the matching of an
extension's `lists` and `rooms` against the response. Phase 5 uses it,
and it is cheap to port now. A list without exactly one `SYNC`
operation fails, as Python's assertion does: that is a list with no
ranges or several, under an extension that names it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room.rs` holds the parts of `get_room_sync_data`
that make no reads:
- the timeline bounds (`from_bound`, `initial`, the expanded timeline)
- `num_live`
- the stripped state of an invite or knock
- the string fields of the name, avatar and member events, and the
hero assembly
- `_get_bump_stamp` split around its two reads, plus the
`always_return_bump_stamp` rule and the clamp to zero
- the room config the connection records afterwards
Three of these move, unchanged, out of `get_room_sync_data` into Python
helpers, so the tests compare against the real code: `_timeline_bounds`,
`_stripped_state` and `_room_config_to_persist`. `_get_bump_stamp` is
tested as it is, with a stand-in store.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
`rust/src/sliding_sync/room_lists/sort.rs` holds `sort_rooms` as a
`SortPlan`: the decision logic with the stream change cache shortcut,
and the caller fetching what each step asks for. `lists.rs` holds the
ranges to `SYNC` operations with the full-range no-sort shortcut, the
config combining across lists and room subscriptions, and the decision
half of `_filter_relevant_rooms_to_send`, including the reset for a
connection with too many rooms to update.
The Python ranges loop moves, unchanged, into `_sync_list_ops`, so the
test can call it. It keeps a quirk the Rust ports: once a room combines
with a config from an earlier list, every later room in the same list
takes the combined config too.
The relevant room map is an `IndexMap`, which keeps Python's dict order,
so `indexmap` becomes a direct dependency. It was already in the tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L51DoxB8VnH8UjXJFDgpNG
