The read paths now report the same spans as the write path:
- `PoolQuerier` (event fetch, stream ID allocation): `db.connection`
from checkout until it drops, and a `db.query` per statement, tagged
`db.prepared` when the statement missed the connection's cache.
- The sync store's borrowed connections: `db.connection` from checkout
until the drop, and for a transaction a `db.txn` (`db.txn_desc` of
`sync` or `sync_write`, `db.txn_id`, `db.readonly`) with a `commit`
log, or a `db.rollback` child when a dropped transaction rolls back.
Each statement is a `db.query`, with `db.begin` when the pending
BEGIN went with it.
- The state resolution store: `db.connection` → `db.txn`
(`state_res`, read only) → a `db.query` each for the BEGIN, every
read and the COMMIT.
As on the write path and in Python, `db.connection` starts once the
connection is in hand, and the wait for it is a `checked out` log on
the caller's span. A failed `db.query` is marked with the `error` tag
and log. `db.query` spans take the held connection or transaction span
as an explicit parent.
The tests collect spans on a current-thread runtime: the registry
releases a span's enter through the thread's default subscriber, so a
span entered on a worker thread would never close under a test's
thread-local subscriber. Production installs the layer globally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
A write interaction on the native driver now reports spans with the
names and tags of Python's `DatabasePool` spans, so Jaeger searches
and per-operation views work across both:
- `db.{name}` around the whole interaction, every attempt and the
waits for a connection included, with a `checked out` log carrying
each wait and a `retry` log per retry;
- `db.connection` per attempt, from having the connection to giving it
back;
- `db.txn` per attempt, tagged `db.txn_desc`, `db.txn_id` (one per
interaction, as Python names it) and `db.attempt`, with a `commit`
log, and a `db.rollback` child when it rolls back;
- `db.query` per round trip, tagged `db.type` and `db.statement`.
Rust retries on a fresh connection, so unlike Python each attempt has
its own `db.connection`.
One `db.query` covers a round trip rather than a statement, since the
round trip is what the pipelining saves. `db.queued` counts the queued
writes sent ahead of the statement and `db.queued_statements` lists
them, `db.begin` says whether the BEGIN went with it, and
`db.prepared` counts statements prepared first, in a `db.prepare`
child for that extra round trip. Statements are only formatted for
sampled spans.
A span an error leaves is marked as opentracing marks one an
exception leaves: an `error = true` tag and an `error` log.
`_load_next_mult_id`, which runs outside `run_write_interaction`, gets
its `db.{name}` span too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
The write transaction read six times between its writes, each read a
round trip: the room's depth, which events it already had, their
redactions, whether the prev events were persisted, the cache prefill's
rejections and redactions, and a cache stream ID. Now the rooms row of a
new room and the depth update are queued writes (the depth update
conditional, as Python's read made it), the three reads it still needs
go out together with eight cache stream IDs in one round trip behind
them, through the new WriteTxn::query_many, and the rest is known: the
cache prefill from the contexts and the redactions already read, and a
prev event the chunk itself persists needs no check. A chunk that
creates a room is written in two round trips instead of eight.
The createRoom bench gets a sane event cache, as the test default of one
entry made every read after persisting a fetch, which no server does.
The persist transaction sent its statements one at a time: about forty
round trips for a new room, about twenty for a message.
Every write whose result nothing reads is now queued: the batch inserts
and upserts, the deletes and updates, the current state and membership
batches, the sliding sync rows, threads and sticky events. Queued writes
go out together ahead of the next read, or with the COMMIT, in the order
the code issues them, so the statements run in the same order as before.
What is left are the reads, each a round trip carrying the writes before
it: the room's share lock, room depth, outliers, redactions, backward
extremities and the cache stream's IDs.
The partial state insert stays immediate: it turns a constraint
violation into `PartialStateConflict`, so the queue is sent first, and a
queued write's own violation cannot be taken for it. `upsert_one`, for
out-of-band memberships, also stays immediate.
`OwnedColumn::copy` and `owned_params!` copy borrowed values into ones
a queued statement can own (`ToOwnedSql`), and `execute_batch_later`
queues a statement per row.
The new room's write transaction goes from 42 round trips to about 12.
The E2EE monitor's room on jki.re goes from 102 to 79 round trips
overall, a plain room from 90 to 63.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
A `WriteTxn` sent each statement and waited for it before the next, so
a transaction of forty writes took forty round trips, though nothing
read most of their results.
Writes can now be queued with `execute_later`, `insert_many_later` and
`upsert_many_later`, which take owned parameters. Every statement that is
not queued, and the `COMMIT`, first sends the queued writes, in order,
in the same round trip: tokio-postgres sends a request when its future is
first polled, and `join_in_order` polls them in order. So no read can
overtake a queued write, and a transaction ending in queued writes sends
them with its `COMMIT`. Statements not yet prepared on the connection
are prepared together first.
tokio-postgres encodes a request's parameters on its first poll and sends
nothing if that fails, while later requests still go out. So the queued
writes' parameters are checked against their prepared statements' types
before any is sent: otherwise the writes after a bad one, and the COMMIT,
would go through. A dry run sends its queue before rolling back, so it
still checks the writes. The statement methods must not run
concurrently, or a statement could overtake writes queued before it.
A queued write that fails surfaces at the next statement, reported as
its own failure (with an optional mapping, for callers that turn a
constraint violation into a domain error), rather than as the "current
transaction is aborted" error of the statements after it. A failed
attempt drops its queue unsent.
`execute_batch` uses `join_in_order` too: `try_join_all` does not promise
to poll long lists in order.
Nothing queues writes yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Every transaction on the native driver sent `BEGIN` on its own and
waited for it before the first statement: one round trip per
transaction that did nothing but open it.
The `BEGIN` now goes out in front of the first statement's first
request: its execution if the connection has the statement cached, its
prepare otherwise. tokio-postgres queues a request when its future is
first polled, so joining the two puts them on the wire in order, and
Postgres runs the statement inside the transaction. This applies to
Python's transactions (`Connection::with_client`, through a
`PendingBegin` each statement method gets) and to Rust's write
interactions (`WriteTxn`, whose first statement claims the `BEGIN`).
If the `BEGIN` fails, the statement behind it may have run outside a
transaction, so the connection is discarded rather than reused. A retry
could then apply that statement twice, but `BEGIN ISOLATION LEVEL ...`
has no realistic way to fail on the server while the next statement
succeeds: a broken connection fails both. A
`WriteTxn` that runs no statement now sends neither `BEGIN` nor
`COMMIT`.
The E2EE monitor's room on jki.re goes from 150 to 135 round trips, a
plain room from 104 to 93.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
The native driver's write path recorded no `sql_txn_count` or
`sql_txn_duration`, nor the log context's transaction counts, so the
`persist_events` panels read near zero at `rust_event_persistence:
transaction` and `full`.
`run_write_interaction` now times each interaction as `new_transaction`
does: every attempt, from when it has its connection, so the wait for
one is left out. The times go into a task-local list that
`with_txn_timings` collects. The persister's entry points carry the list
back on what they already return (`PersistEffects`, `PreparedChunk`,
`PersistedBatch`, `CurrentStateUpdate`), and Python records it where
`new_transaction` would: the two counters, the pool's perf counters and
the current log context, which is the caller's when Python applies what
Rust returned. That costs no GIL hop.
When a persist fails, or Python gives up a batch, no result carries the
times back, so the persister keeps them and `flush_rust_persist_metrics`
counts them at the next scrape, without a log context. In shadow mode
Python's own interactions are the ones used, so Rust's dry run is not
recorded. The stream orderings' `nextval`, which runs outside a write
interaction, is recorded as `_load_next_mult_id`, as Python's is. The
`db.<desc>` tracing span is not recorded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The first commit of phase 2 of the event persistence plan
(`transaction` level). `rust/src/persist/txn/` will hold the port of
`_persist_events_txn`. This commit adds its driver: the share lock on
the room's `rooms` row, the cache updates Python queues for every event
up front, and dropping duplicate events. It returns the events left for
the steps still in Python.
`tests/storage/test_rust_persist_txn.py` builds the same DAG in two
rooms. Room A persists in Python. Room B prepares in Rust, runs the
Rust part of the transaction in its own committed transaction through
`Persister.persist_events_txn_for_test`, and then runs the Python steps
after it. The test compares the rows of every table the transaction
writes, after normalising room and event IDs, stream positions, state
groups and chain IDs, and compares the cache updates both rooms asked
for. Each later commit moves more steps from the Python half to the
Rust half, in transaction order, until the facade runs the whole
transaction. Mutating the driver (dropping the redaction
invalidation, the relation, the state key, or the outlier rule for
duplicates) fails the comparison.
Two open items from phase 1:
- Reads before the transaction now retry. Python runs each read in a
`runInteraction` that retries a lost connection or a cancelled
statement. `Persister.prepare_chunk` now prepares the whole chunk
again on such an error, up to five attempts. That is safe because its
only write, bumping the state groups it uses, can be repeated. The
reader and the state resolution store keep the pool's error as the
cause, so `is_retryable_error` can classify it. A `WriteError` does
not count, as its write interaction has already retried, and
`WriteError::from_pg` shares the classification.
- The Rust persister needs the state tables in the main database, as
it reads and writes both in one transaction. `Persister::for_homeserver`
now checks for that, and the persister is off when they are apart.
Also fixed in the dispatch: at `off`, `_prepare_chunk` ran Rust when a
persister existed. That only happens when a test changes the level at
runtime. The twin test does exactly that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`Persister.prepare_chunk` runs the steps of the previous commits for a
chunk: the extremities and state delta, then the sliding sync table
changes and sticky events, then the chain cover index. The chain cover
step comes last as the only one that writes, so a failure in an earlier
step leaves nothing for Python to find. Sliding sync and sticky events
now come before the stream orderings are allocated, as nothing there
reads them. Each step reads the clock when it needs the time.
`_persist_event_batch` dispatches on `rust_event_persistence`:
- `shadow`: Rust runs first with its chain cover write rolled back, then
Python. The results are compared and Python's are used. The comparison
ignores orders Python leaves to chance and the IDs of new chains. Both
rejecting the chunk with the same error is a match.
- `prepare` and above: Rust's results are used. A `SynapseError` is
raised as Python would raise it. Python computes the chunk when Rust
cannot resolve state in the room, or on any other Rust error.
Trial runs strict by default (`SYNAPSE_TEST_RUST_PERSIST_STRICT`), so a
mismatch or Rust error fails the test. Rust's counts and state res stats
are recorded by Python once it uses the result.
At `prepare`, the `_calculate_state_and_extrem` and
`calculate_chain_cover_index_for_events` timings give way to
`rust_prepare_chunk`. Python's state resolution and state group caches
are not filled, so creating a room costs one more transaction
(`RoomsCreateTestCase` counts it) and the first local event after a
fork resolves state again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`PersistReader` wraps a `SyncReader` and adds the reads the persist
controller makes before its write transaction: forward and state DAG
extremities, the prev event walks of `_get_events_which_are_prevs` and
`_get_prevs_before_rejected`, `have_seen_events`, memberships by event ID,
`is_local_host_in_room_ignoring_users`, the room version, the sliding sync
joined room check, `get_last_event_pos_in_room` and the soft failed sticky
events. None of them is cached. Each runs once per chunk.
The chain cover reads are free functions over a `Querier`, not reader
methods. The chain cover step reads, allocates chain IDs and writes in one
transaction, so they must run on that transaction's connection. To allow
that, `WriteTxn` now implements `Querier`. A `WriteError` that passes
through `anyhow` on the way out comes back as itself, so a conflict is still
retried.
The persister builds its own `SyncStore` over the shared caches.
`tests/storage/test_persist_reader.py` runs each read against the Python
method or statement it ports, on the native driver.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
`run_write_interaction` is the Rust `runInteraction` for writes: a
`REPEATABLE READ` transaction, committed at the end, and run again from the
start on a serialization failure, a deadlock, a cancelled statement or a
lost connection, up to five attempts as in Python. Constraint violations come
back as their own error, so an expected conflict can be told apart.
Once a statement fails, Postgres has aborted the transaction, and a COMMIT
would roll back without an error. So if the closure catches a statement
error and returns success, the commit is refused with an error.
`WriteTxn::insert_many` and `upsert_many` write a batch in one statement
whatever its size, passing each column as an array to `UNNEST`.
`execute_batch` pipelines a per-row statement for the inserts that need a
sub-select per row.
The tests run against a real server when `SYNAPSE_TEST_POSTGRES_DSN` is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
