`Persister.prepare_chunk` runs the steps of the previous commits for a
chunk: the extremities and state delta, then the sliding sync table
changes and sticky events, then the chain cover index. The chain cover
step comes last as the only one that writes, so a failure in an earlier
step leaves nothing for Python to find. Sliding sync and sticky events
now come before the stream orderings are allocated, as nothing there
reads them. Each step reads the clock when it needs the time.
`_persist_event_batch` dispatches on `rust_event_persistence`:
- `shadow`: Rust runs first with its chain cover write rolled back, then
Python. The results are compared and Python's are used. The comparison
ignores orders Python leaves to chance and the IDs of new chains. Both
rejecting the chunk with the same error is a match.
- `prepare` and above: Rust's results are used. A `SynapseError` is
raised as Python would raise it. Python computes the chunk when Rust
cannot resolve state in the room, or on any other Rust error.
Trial runs strict by default (`SYNAPSE_TEST_RUST_PERSIST_STRICT`), so a
mismatch or Rust error fails the test. Rust's counts and state res stats
are recorded by Python once it uses the result.
At `prepare`, the `_calculate_state_and_extrem` and
`calculate_chain_cover_index_for_events` timings give way to
`rust_prepare_chunk`. Python's state resolution and state group caches
are not filled, so creating a room costs one more transaction
(`RoomsCreateTestCase` counts it) and the first local event after a
fork resolves state again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`EventsWorkerStore._get_event_cache` is now the Rust `EventCache` that the
sync handler and state resolution read, so each event is held once.
The cache keeps Rust `CachedEvent` values. Python gets them through
`RustEventCache`, a `RustCache` subclass with the `AsyncLruCache` names the
rest of Synapse uses, and each hit wraps the cached event in a new `Event`
object. The wrapper shares the cached event's internal metadata, unsigned
data and signatures, so a change Python makes, such as adding
`prev_content` or flipping `outlier`, lands in the one copy the Rust
readers see. Python's `_event_ref` weak map is checked before the cache,
so while anything holds an event, a lookup returns that same object, as
before. A hit there marks the cache entry as read, and puts the event back
if it was evicted.
- The cache is sized from `caches.event_cache_size` and named
`*getEvent*`, so its metrics and `per_cache_factors` keep their names.
The Rust loader now counts its misses too.
- Invalidating an event drops its chain position as well, and a room
purge drops the room's events through the room index, plus the chain
cover and state group caches, as `RustStore.invalidate_event` and
`clear_event_cache` did. Those two calls are gone.
- `EventsWorkerStore` and `RustWorkerStore` run their `__init__` bodies
in a different order in `DataStore` and `GenericWorkerStore`, as they
sit differently in each MRO. So whichever runs first builds the
`RustStore`. A datastore without `RustWorkerStore`, such as the port
script's, gets a standalone `RustEventCache`.
- The external cache stubs (`get_external`, `set_external`,
`_invalidate_async_get_event_cache` and its room variant) are dropped.
They were documented no-ops.
- The sync shadow diff still ignores `prev_content` on state events.
Python keeps the objects it holds after the cache evicts them, so a Rust
reader can still load a second copy without it. Trial's cache of one
event makes that routine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
With the database work cached, converting the state sets from Python
dicts was the largest piece of a warm resolution: 5,010 entries pulled
out of tuples and interned, under the GIL, before the future starts,
and 2,505 entries built into a dict on the way back. Every group's
state was converted again on every resolution it took part in.
The handler gains resolve_state_groups, which takes the state groups
by ID with their state, as StateResolutionHandler.resolve_state_groups
receives them, and keeps each group's converted StateMap in a cache
keyed on the group ID. State groups are immutable and their IDs are
never reused, so an entry can only go stale by its group being
deleted, which makes it dead weight rather than wrong. The cache is
cleared on a room purge with the others. A group given with fewer
entries than its cached state is a filtered one: it is converted for
that call and does not replace the full map.
The result is returned in the shape of a _StateCacheEntry, chosen as
_make_state_cache_entry does: the input group the resolved state
equals, else the input group it differs least from and the difference,
else the state itself. So the common cases cross the boundary as a
group ID or a small delta rather than the whole state.
StateResolutionHandler.resolve_state_groups uses it in "on" mode, with
the usual fallback to Python for rooms without a chain cover index.
Shadow mode keeps comparing through resolve_events_with_store. The
state sets are now shared maps inside the Rust, so a cached map and a
converted one are handled alike.
On the benchmark room (2,000 members, 1,000 conflicted events) a warm
resolution through the state groups takes 9 to 11 ms end to end once
the groups' state is cached, from 15 to 19 through the flat entry
point; Python warm is 37 to 44.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
With the event cache warm, the chain cover queries were what was left of
the database work in a resolution: the position of every event in the
state sets, and the links out of every chain they are on. Both can be
cached.
A chain position never changes once assigned, so it is cached before
the transaction commits, and dropped with the event in case the event is
purged and received again. The links out of a chain only grow, at the
end: an event appended at sequence number n adds links out of (chain, n)
and changes nothing below. So links read while the chain was known to
reach n are complete for every walk that starts at or below n, and the
cache keeps that bound with them. A walk that needs a chain beyond its
bound refetches it. Both caches go on a room purge, which deletes the
room's chains and links.
The chain cover walk now gathers links first (from the cache, following
the links a backwards walk would follow so reached chains are gathered
too, and from the database for the rest) and then walks them once. The
Python walks each batch of its recursive query as it arrives, but a batch
is closed under reachability, so walking the union comes to the same
thing. Fetching only what a sequence-aware walk reaches, rather than the
full reachability closure, leaves the auth difference and the v2.1
conflicted subgraph unchanged: any forwards path that ends inside a
backwards-reachable range runs through chains the backwards walks
reached, below their bounds. The trial comparison against the Python
runs every case through a handler's caches too.
The three store-side caches are bundled as StateResCaches, so the store
and the invalidation hooks treat them as one. The integer-keyed maps in
the chain cover code and the interner use FxHash, and the reverse
position map is one flat map, which together are worth 2 ms on the warm
path of the benchmark. The stats gained the cache hit counts and
timings for the auth chain difference, event loading and the
connection.
On the benchmark room (2,000 members, 1,000 conflicted events) a warm
resolution runs no chain queries and takes 14 to 20 ms end to end, from
27 to 32 with the event cache alone; Python warm is 38 to 41.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
Loading and parsing events was most of a Rust resolution's time: against
a warm Python event cache, Rust was level with Python. The handler now
keeps the events it loads, keyed by event ID, so a resolution in a room
it has seen reads them from memory.
The conflict cache's LRU becomes a generic SizedLru shared by both
caches. The Postgres store takes the event cache, reads it before the
database and inserts what it loads. The stats report event cache hits
and the time spent converting state to and from Python.
Invalidation follows the Python event cache rather than having its own
sources: EventsWorkerStore._invalidate_local_get_event_cache and the
room-wide variant drop the same events from the Rust cache. Those two
methods are where every event cache invalidation ends up, including a
partial-state event's rejection status changing, which is the one
change state resolution can observe.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
For v2 and v2.1 rooms on the native driver:
- "on" resolves in Rust, falling back to Python for rooms without a
chain cover index. The per-room CPU and database metrics come from
the Rust stats, as the log context does not see that work.
- "shadow" resolves in Python, then in Rust with a copy of the event
map, logs any difference and counts the outcome in
synapse_state_res_rust_shadow_total.
- "off" (the default) is unchanged.
SYNAPSE_TEST_RUST_STATE_RES sets the mode for every test homeserver.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
