The queries of the per-room sync loop, on the native Postgres driver, each
a port of the store method it is named after. A `SyncReader` is made per
request and borrows a pooled connection for one group of statements at a
time, so that a sync worker with hundreds of requests in flight does not
pin a connection per room; a group that needs one snapshot runs in a
`REPEATABLE READ READ ONLY` transaction. State groups are loaded whole,
with the recursive CTE sent as one simple query per batch, and cached as
full maps.
The caches that copy a `@cached` function (rooms for user, room summary,
partial state rooms, erased users, retention policies, event to state
group, unread counts) live on the Rust store and are attached to the
Python cache each names by the new `RustWorkerStore`, which also takes
over building the Rust store. Unread counts are invalidated by room as
well as by room and user, as the Python tree cache is.
`tests/storage/test_sync_store.py` runs every query and the Python
method against the same database, with multi-writer tokens, redactions
of every kind, and the summary rotation of the unread counts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
The first of the pure parts of `/sync` that the Rust sync handler needs:
`RoomStreamToken` with its three string forms and its vector clock
arithmetic (`is_before_or_eq`, `copy_and_advance`, `bound_stream_token`),
`persisted_after`, and the pagination helpers of `stream.py` that turn
tokens into bounds and filter the rows that come back by writer. The SQL
is built in the Postgres row-comparison form only, as the handler reads
through the native Postgres driver.
Tokens cross the boundary as the Python class, converted both ways.
The multi-writer filtering is where a bug would be silent, so
`tests/sync/test_rust_stream_token.py` checks every operation against the
Python on random tokens with instance maps. `tests/sync/fixtures.py`
holds what the differential tests that follow share.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194eNQP5weBtVMg8NKtc6VU
### Summary
This moves the synchronous core of client event serialization out of
`synapse/events/utils.py` and into Rust
(`rust/src/events/serialize.rs`).
Event serialization is on the hot path for `/sync`, `/messages`, and
most client-facing endpoints. Previously it was a recursive pure-Python
routine (`_serialize_event` / `_inject_bundled_aggregations` /
`only_fields`) that interleaved
CPU-bound formatting with `async` DB/IO. This PR separates those two
concerns and moves the CPU-bound half to Rust:
- **Async prep stays in Python.**
`EventClientSerializer._prepare_serialization` does all DB/IO up front:
batch-fetching redaction events and running the registered module
`unsigned`-callback hooks, for the top-level events *and* every bundled
sub-event (edits and thread latest events, which are themselves
serialized). The admin/MSC4354 config is resolved once via
`_update_config`, rather than re-checked on every recursive call as the
old code did.
- **The synchronous core moves to Rust.** Given an event plus the
pre-fetched `redaction_map`, `unsigned_additions`, and
`bundle_aggregations`, the Rust code produces the client JSON entirely
in Rust — including the v1/v2 format transforms, `only_event_fields`
filtering, redaction handling, and recursive bundled aggregations.
### Details
- The Rust entry point is a single batch function, `serialize_events`,
taking a list of `(event, membership)` pairs. The three lookup maps are
shared across the whole batch, so they're read out of Python and
converted to Rust
structures **once** per batch rather than once per event.
`EventClientSerializer.serialize_event` (singular) is a thin wrapper
that calls it with a one-element list.
- `SerializeEventConfig` is now a Rust `pyclass`, and the old
`event_format` callable is replaced by the `EventFormat` enum (`Raw` /
`ClientV1` / `ClientV2` / `ClientV2WithoutRoomId`). Call sites in
`rest/admin/events.py` and
`rest/client/{notifications,room,sync}.py` are updated to pass the enum.
`make_config_for_admin` and MSC4354 enablement now go through
`SerializeEventConfig.for_admin()` / `with_msc4354()`.
- New accessors on `EventInternalMetadata` (`redacted_by`, `txn_id`,
`device_id`, `token_id`, `delay_id`, `soft_failed`,
`policy_server_spammy`) expose to Rust the fields the serializer reads.
- The `_split_field` unit tests move from `tests/events/test_utils.py`
to a Rust test in `serialize.rs`, since the implementation moved.
### Behaviour
This is intended to be a behaviour-preserving refactor — the Rust core
mirrors the previous Python output (field ordering, v1 key promotion,
redaction placement per room version, null-`redacts` handling,
transaction-ID gating).
Existing serialization, relations, and sync tests pass unchanged.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Eric Eastwood <erice@element.io>
This is in prep for converting the event serialization to Rust.
This is a fairly mechanical port, except that we store the appservice ID
rather than the appservice object. This avoids us having to store a
`Py<..>` (or port the appservice object over).
