Reading a `json`/`jsonb` column failed with "error deserializing column" — the
value mapping had no decoder for them. Synapse reads genuine `jsonb` values, most
visibly for custom profile fields (`get_profile_field` selects
`JSONB_EXTRACT_PATH(fields, ?)`), so those requests returned HTTP 500 on the Rust
backend.
Decode `json`/`jsonb` into the matching Python object (dict/list/scalar), as
psycopg2 does by default: `json` is the raw JSON text, `jsonb` a one-byte version
header (currently 1) then the text; parse with serde_json and convert via
pythonize. `PythonPgFromSql::accepts` now also accepts `json`/`jsonb`; the
Rust-native `DbValue` decoder stays scalar-only (nothing reads json through it).
Fixes the profile custom-field 500s (e.g. test_non_string,
test_can_lookup_own_profile) on the Rust backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
The value mapping bound arrays as parameters (a `list` → `Array`) but couldn't
decode an array *column* back out — the decode side assumed "Synapse only binds
arrays as parameters". That assumption was wrong: the caches replication stream
reads `keys` (a `text[]`) via `get_all_updated_caches`, so decoding failed with
"error deserializing column", the stream read errored, and a worker never
received bulk cache invalidations (the `wait_for_stream_position` in the test
hung and never fired).
Decode an array column into a Python `list`, each element decoded by the array's
element type (via `array_from_sql`); `PythonPgFromSql::accepts` now also accepts
arrays of a supported scalar element type, mirroring `PgValue`'s `ToSql`. The
Rust-native `DbValue` decoder stays scalar-only — nothing reads array columns
through it. Adds `fallible-iterator` (the version postgres-protocol already
uses) to iterate the decoded array's elements.
Fixes tests.storage.databases.main.test_cache.CacheInvalidationOverReplication's
test_bulk_invalidation_replicates on the Rust backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
`adbapi.ConnectionPool` obtains connections via a `connectionFactory` attribute,
which the database-outage tests swap out to make every checkout fail (and
restore afterwards). `RustConnectionPool` checked out connections inline, so it
had no such hook and the outage tests errored with `AttributeError`.
Give it a `connectionFactory` attribute (defaulting to a checkout from the
native pool, wrapped in the DBAPI2 adapter) and route both `runWithConnection`
and `connect` through it, so a test's replacement takes effect. Also make the
native pool reopenable: `close` drops it and `start` opens a fresh one, so the
tests' `close()`/`start()` outage cycle recovers rather than leaving the pool
permanently shut.
Fixes the tests.storage.databases.main.test_events_worker DatabaseOutage
failures on the Rust backend (12/12 pass); psycopg2 and sqlite are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Two value-mapping gaps surfaced by the storage suite on the Rust backend:
- `bytearray` parameters. Synapse deliberately passes binary data as
`bytearray` (the psycopg2 engine disables the `bytes` adapter to catch
accidental text-as-bytes bugs), but the shim only accepted `bytes`, so
binary background-update parameters raised `TypeError`. Accept `bytearray`
as a BYTEA parameter too.
- `tid` columns. The receipts-dedup background update selects a row's `ctid`
and then compares against it (`WHERE ctid != ?`). psycopg2 renders `tid` as
a `(block,offset)` string and accepts that string back; the shim had no
`tid` support at all. Decode `tid` to that same string (three big-endian
u16s: the block number's halves, then the offset — matching the server's
`tidsend`) and encode the string back to the wire form when it is bound as a
parameter.
Both directions are covered for the Python (`PythonPgFromSql`) and the
Rust-native (`DbValueFromSql`) decoders and the `ToSql` encoder, and `tid` is
added to the shared `accepts_column_type` set.
Fixes tests.storage.test_user_filters.test_bg_migration and
tests.storage.databases.main.test_receipts's linearized-unique-index background
update on the Rust backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
`new_transaction` sets a transaction's isolation level (for the receipts,
push-actions and purge paths, which ask for a specific level) by calling
`engine.attempt_to_set_isolation_level` before the transaction and resetting it
after. The Rust engine raised `NotImplementedError` there, so those transactions
errored out on the Rust backend.
The shim has no psycopg2-style `conn.set_isolation_level`, so implement it in
SQL: run `SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL <level>` on
the connection. That sets the session's default for *subsequent* transactions,
so it's committed before the caller's transaction begins (SET is transactional);
`new_transaction` resets it to the default (REPEATABLE READ, which the pool
already applies at connection setup) afterwards, scoping the override to the one
transaction — matching psycopg2's behaviour. The level name comes from a fixed
`IsolationLevel` map, not caller input.
Fixes the isolation-level failures in tests.storage.test_event_push_actions,
tests.storage.test_purge and tests.storage.test_receipts on the Rust backend
(20/20 pass); psycopg2 and sqlite are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
`RustConnectionPool` stands in for `DatabasePool._db_pool`, a
`twisted.enterprise.adbapi.ConnectionPool`. It presented the slice Synapse's
`DatabasePool` uses (`runWithConnection`, `threadID`), but some tests reach for
`_db_pool` directly and call adbapi's `runQuery` / `runOperation` / `connect`,
which weren't implemented — so those tests errored with `AttributeError` on the
Rust backend.
Add them, mirroring adbapi's semantics:
- `runQuery` / `runOperation` run one statement in its own transaction on a
worker thread (commit on success, roll back on error), returning the rows
(or nothing) as a `Deferred`.
- `connect` returns a connection for direct synchronous use on the caller's
thread, cached per thread as adbapi does and closed together with the pool.
Fixes the tests.storage.test_appservice and tests.storage.test_rollback_worker
failures on the Rust backend (26/26 pass); psycopg2 and sqlite are unaffected
(they use the real adbapi pool).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
The `populate_user_directory` background update stored its stream position in a
`TEXT` column (`_temp_populate_user_directory_position.position`) but wrote an
int into it and read it back into the `BIGINT`
`user_directory_stream_pos.stream_id` — the one place in the storage layer that
bound an int to a text column (and a numeric string back to an int column). This
only worked because psycopg2 coerces such literals; the native Rust driver binds
typed parameters and rejects the mismatch.
Fix it at the source: make the temp column `BIGINT`, matching the value it holds
(and `update_user_directory_stream_pos`'s `int` type hint, which the `TEXT`
column had been quietly violating). The temp table is created and dropped within
the background update, so there's no migration.
Tested: user_directory storage + handler tests pass on all three backends
(psycopg2, sqlite, Rust).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Add SYNAPSE_TEST_RUST_DRIVER to run the Postgres test suite against the native
Rust driver (sets use_rust_driver on the test database config, off by default).
Teach the harness's make_fake_db_pool to build a RustConnectionPool for the Rust
engine, with its worker thread pool swapped for the threadless test pool so
queries run deterministically on the reactor during `pump` (the shim's block_on
waits there for the tokio workers). Test-database create/drop is admin work, so
it now goes directly over psycopg2 (always available in tests) rather than the
engine module's `connect`.
Running the suite this way surfaces a few real psycopg2-compatibility gaps:
- `fixup_config_defaults` (was `fixup_default_host`) now also fills libpq's
PGUSER / PGPASSWORD environment defaults, not just the host — tokio-postgres
reads none of them from the environment, but Synapse's test setup relies on
them (as psycopg2 does).
- `build_dsn` skips `None`-valued args, matching psycopg2's handling of unset
kwargs.
With these, a homeserver boots on the Rust backend under the test harness and
starts running real transactions and background updates. The remaining psycopg2
type-coercion / behaviour differences are addressed in the commits that follow;
psycopg2 and sqlite runs are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
`DatabasePool`'s `inner_func` calls `conn.reconnect()` when a connection is
found closed, or to recycle one that has exceeded the per-connection
transaction limit (`txn_limit`). The Rust DBAPI2 adapter had no `reconnect`, so
those paths (off by default, but real) would have raised.
Add `reconnect` to the adapter: it returns the current connection to the pool
(or discards it if unusable) and checks out a fresh one. The adapter now holds
the pool it was checked out of, with an `owns_pool` flag distinguishing a shared
pool (RustConnectionPool — reconnect from it, don't close it) from a bootstrap
pool-of-one (`rust_dbapi.connect` — closed together with the connection).
Tested: reconnect swaps in a working connection; runWithConnection and the
bootstrap/adapter paths are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
A running homeserver exercises three shim capabilities beyond what the unit
tests reached; add them so it boots on the Rust backend end to end:
- `prepare_database` reads `db_conn.autocommit`; add a readable `autocommit`
getter to the shim `Connection` (and the adapter), mirroring psycopg2's
property.
- schema preparation runs multi-statement scripts via
`engine.execute_script_file` -> `cursor.executescript`; route
`LoggingTransaction.executescript` to the shim's multi-statement primitive
for the Rust backend (as it already does for sqlite), and expose
`executescript` on the DBAPI2 adapter cursor.
- store loading binds Python lists as Postgres arrays for `column = ANY($1)` /
`!= ALL($1)` (Synapse's `make_in_list_sql_clause`); add a `PgValue::Array`
variant that classifies a `list` and encodes it as an array of the element
column type, and accept array column types in `ToSql`.
With these, a homeserver boots on the Rust backend: it prepares the full schema
(176 tables), listens, `/health` returns OK, and DB-backed client endpoints
respond correctly.
Tested: array `to_sql` / classification and array-column `accepts` (Rust unit
tests); and, against a live Postgres, `= ANY($1)` list binding, adapter
`executescript`, and the `autocommit` getter. sqlite and psycopg2 homeserver
boots remain green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Turn the Rust backend on with `database.use_rust_driver: true` (alongside
`name: psycopg2`). Keeping `name: psycopg2` means the engine is still selected as
Postgres and all the `isinstance(engine, PostgresEngine)` dialect checks hold;
the flag only swaps the driver implementation.
- `create_engine` returns `RustPostgresEngine` instead of `Psycopg2Engine` when
the flag is set.
- `make_pool` gains a `clock` argument and, for the Rust engine, builds a
`RustConnectionPool` (via `_make_rust_pool`) instead of an adbapi pool:
it derives the libpq DSN from the config `args`, sizes threads/connections
to `cp_max`, passes the engine's synchronous_commit / statement_timeout,
starts the pool, and registers a shutdown hook via the clock. Its declared
return type stays `adbapi.ConnectionPool` — `RustConnectionPool` provides the
`_db_pool` subset `DatabasePool` uses (runWithConnection / threadID /
running / threadpool) — so callers that lean on adbapi-specific methods keep
type-checking.
- `RustConnectionPool` takes synchronous_commit / statement_timeout_ms and
threads them into its pooled connections' session setup.
- `DatabasePool` passes its clock to `make_pool`; the test harness's
`make_fake_db_pool` accepts the new argument.
Tested: create_engine honours the flag (and the Rust engine is still a
PostgresEngine, not a Psycopg2Engine); make_pool builds a started
RustConnectionPool that serves a query as `_db_pool`. psycopg2 and sqlite
homeserver boots remain green. A full homeserver boot on the Rust backend isn't
covered here because the test harness patches `make_pool` with a synchronous
adbapi fake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Startup opens a bootstrap connection via `make_conn` (outside the runtime pool)
and validates it with `engine.check_database` before `prepare_database` runs.
Neither worked for the Rust backend: the Rust module has no `module.connect`,
and check_database/server_version were NotImplementedError stubs. Fill them in:
- `rust_dbapi.build_dsn` turns the libpq-style `args` into a DSN string, and
`rust_dbapi.connect` opens a standalone connection (a pool of one, kept
alive by the returned Connection) for one-off bootstrap use.
- `make_conn` routes the Rust engine through `rust_dbapi.connect` (with the
engine's synchronous_commit / statement_timeout) instead of
`engine.module.connect`; psycopg2 and sqlite are unchanged.
- `RustPostgresEngine.check_database` reads the server version over a cursor
(`SHOW server_version_num`) rather than psycopg2's `conn.server_version`,
and applies the same version / encoding / collation / ctype checks;
`server_version` is derived from the cached value.
Tested: build_dsn quoting; and, against a live Postgres, make_conn +
check_database + server_version + a query. psycopg2 and sqlite homeserver boots
(make_conn → check_database) remain green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Make RustConnectionPool a drop-in for DatabasePool._db_pool so real
runInteraction traffic can flow through it (once make_pool is switched):
- it now hands each function a DBAPI2-adapter connection (rust_dbapi.Connection)
wrapping the pooled shim, so LoggingDatabaseConnection.cursor() yields a
working LoggingTransaction and the engine's in_transaction / is_closed /
set_autocommit operate on it;
- the entry point is named `runWithConnection` (matching
twisted.enterprise.adbapi.ConnectionPool, which database.py calls by that
name), alongside `threadID` for the transaction-limit path. With
`threadpool` and `running` already present, the pool covers the slice of the
adbapi interface DatabasePool uses.
A new test drives a full transaction through the pool the way
DatabasePool.runWithConnection's inner_func does — engine.in_transaction check,
LoggingDatabaseConnection + LoggingTransaction, `?`→`$n` conversion, commit —
off the reactor thread, and gets the result back via the Deferred.
Still outstanding before make_pool can return it: `reconnect` on the connection
(transaction-limit / closed-connection paths) and the startup path
(make_conn / check_database).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Previously RustPostgresEngine subclassed the psycopg2 PostgresEngine, which
dragged psycopg2 into the Rust path and conflated "is Postgres" with "is
psycopg2". Split the hierarchy so both drivers are siblings:
- `PostgresEngine` (new `postgres_base` module) is now the driver-agnostic
base holding the shared SQL-dialect and config logic (single_threaded,
supports_using_any_list, row_id_name, get_db_locale, check_new_database,
lock_table, synchronous_commit / statement_timeout). It has no driver
dependency, so it always imports. Everything touching a live connection,
the DBAPI2 exception module, or the placeholder style is left abstract.
- `Psycopg2Engine(PostgresEngine)` holds the psycopg2 specifics (register_type
/ register_adapter, isolation-level map, conn.status/closed/server_version,
`%s` placeholders, psycopg2 execute path, `uses_psycopg2_extras = True`).
- `RustPostgresEngine(PostgresEngine)` is re-parented onto the base (no longer
inherits psycopg2). It passes the Rust DBAPI2 module to the base and gets
NotImplementedError stubs for the still-psycopg2-shaped check_database /
server_version (part of the deferred startup wiring).
The base keeps the name `PostgresEngine`, so all ~91
`isinstance(engine, PostgresEngine)` checks across the storage layer (which mean
"emit Postgres SQL") hold for both drivers unchanged. `create_engine` now
returns `Psycopg2Engine` for `name == "psycopg2"`.
Verified: full lint clean; Rust engine/adapter tests pass; a psycopg2 homeserver
boots and runs (test_room_search under Postgres) and the sqlite path is
unaffected (test_room_search / test_event_federation under sqlite).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Synapse's LoggingTransaction drives a cursor through the DBAPI2 spelling
(fetchone/fetchmany/fetchall, iteration, rowcount/description as properties),
but the Rust shim cursor exposes fetch_one/fetch_all/fetch_next_batch and
rowcount()/description() as methods. Rather than reshape the Rust API, add thin
Python `Connection`/`Cursor` wrappers that present the DBAPI2 shape and delegate
to the shim:
- `Cursor` maps fetchone/fetchmany/fetchall/__iter__ and rowcount/description
onto the shim, and tracks exhaustion so fetching past the end keeps
returning "no more rows" (the shim raises instead);
- `Connection.cursor()` returns the adapter cursor, and the transaction-control
and engine-facing methods (commit/rollback/close, set_autocommit, is_closed,
in_transaction) delegate straight through.
An end-to-end test drives a real LoggingTransaction backed by the adapter and
RustPostgresEngine: `?` placeholders are converted to `$n`, a query runs, and
rows come back via fetchone/iteration/description.
Not handled yet: execute_batch / execute_values (psycopg2 extras that
LoggingTransaction calls directly for PostgresEngine) still need a routing
change to reach a shim-backed implementation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Move the equivalent of the engine's `on_new_connection` into the pool's
`ConnectionManager::create`, so every pooled connection is configured once when
it is opened rather than on each checkout. A new `SessionConfig` (carried by the
manager) drives it:
- the default isolation level is set to REPEATABLE READ, matching the engine's
`default_isolation_level`, so a plain BEGIN behaves as it did under psycopg2;
- `synchronous_commit` is turned off when configured;
- `statement_timeout` is set when configured.
`bytea_output` is deliberately not set: tokio-postgres uses the binary protocol
for prepared statements, so that text-format GUC is irrelevant to how we decode
bytea.
`create_pool` keeps its signature (default session); a new
`create_pool_with_session` and two new keyword-only `ConnectionPool(...)`
arguments (`synchronous_commit`, `statement_timeout_ms`) expose the settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
A database engine that drives the Rust Connection/Cursor shim instead of
psycopg2. It subclasses PostgresEngine to reuse the pure SQL-generation and
configuration behaviour, and overrides only the parts that touch a live
connection or are wired to psycopg2 internals:
- `module` points at the Rust backend's DBAPI2 exception hierarchy, which the
transaction driver catches on;
- `convert_param_style` rewrites `?` to `$1, $2, ...` (the shim binds libpq
positional placeholders, not psycopg2's `%s`);
- `in_transaction` / `is_connection_closed` / `attempt_to_set_autocommit` call
the shim's own methods;
- `is_deadlock` matches the Rust `DatabaseError` and its `pgcode`;
- `executescript` uses the shim's multi-statement primitive;
- `on_new_connection` is a no-op — session setup belongs in the Rust pool.
Per-transaction isolation-level overrides raise NotImplementedError for now, and
`check_database` / `server_version` still read psycopg2 attributes; the engine
is not yet selectable via `create_engine`, so neither is reached. Wiring it into
`make_pool` (which also feeds session config to the pool) is the next step.
Tested directly: param-style rewriting, deadlock/pgcode matching, the module
pointer, and — against a live shim connection — in_transaction, is_closed and
autocommit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
The database engine's `is_connection_closed` and `in_transaction` checks (used
by the transaction driver before reusing a pooled connection) need to read
these off the raw connection, as they do from psycopg2's `conn.closed` /
`conn.status`. Add the two accessors to the shim: `is_closed()` is true once the
connection is closed/returned/discarded or the socket is torn down, and
`in_transaction()` reports the tracked transaction flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
`RustConnectionPool` runs synchronous Synapse transaction functions off the
reactor thread against the native Rust connection pool. Each
`run_with_connection(func, *args)` call runs, on a dedicated Twisted thread
pool, `func(conn, *args)` with a `Connection` checked out of the Rust pool, and
returns a Deferred that fires on the reactor with the result (or an errback if
it raised). It builds on `defer_to_threadpool`, so log contexts are preserved
across the hop, and returns the connection to the pool afterwards (clean →
reused, mid-transaction/broken → discarded, per the shim's disposal rules).
The Rust pool sizes 1:1 with the thread count, and `PyConnectionPool.close()`
(new) closes all idle connections so the owner can drop the server connections
deterministically. The bridge is lifecycle-owner-managed (start/close) and
takes no clock, keeping it trivially testable; slotting it into
`DatabasePool.make_pool` — which also needs engine-level support for the shim
connection (in_transaction, is_connection_closed, autocommit/isolation,
reconnect) — is left to a follow-up.
Tested against a live Postgres over the real reactor (Synapse's in-memory test
reactor mocks the DB thread pool out, so a plain Twisted trial TestCase is used):
result/return-value, arg forwarding, exception-to-errback, connection reuse
across calls, and concurrent checkouts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
The Python-facing `Connection`/`Cursor` shim now always wraps a connection
checked out of the `deadpool` pool. `ConnInner` holds a `PooledConnection` and
disposes of it correctly when the last reference (the `Connection` and all its
cursors) goes away, via a `Drop` impl plus `release`/`discard` helpers:
- a clean connection is returned to the pool for reuse;
- a connection is *discarded* (detached with `Object::take`, which also
shrinks the pool) whenever reuse would be unsafe — a failed COMMIT/ROLLBACK,
a poisoned mutex, or being dropped with a transaction still open (which
can't be rolled back synchronously from `Drop`, so the socket close makes
the server do it).
A plain query error still does *not* throw the connection away — it stays
open+aborted for the driver to `rollback()`, exactly as psycopg2 behaves.
The pool is the only way to obtain a connection: the standalone `connect(dsn)`
free function is replaced by a Python-facing `ConnectionPool` class (Rust
`PyConnectionPool`, exposed as `postgres.ConnectionPool`). Build it once from a
DSN, then check connections out with `pool.connect()`. Checkout failures map
onto the DBAPI2 hierarchy — a backend connect error reuses the query-error
mapping (and its `pgcode`), while a timeout / closed pool becomes
`OperationalError` — so `connect()` behaves like psycopg2's for callers.
Adds live-Postgres tests (gated on SYNAPSE_TEST_POSTGRES_DSN) asserting which
connections end up back in the pool, and drives the Python test suite through a
pool in `setUp`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3G4M92AmwSSZCbmtMJU3d
Introduce a small enum-based layer so simple queries can be written once
regardless of backend: DbPool/DbConn dispatch over each backend's own
pool, and DbConn::query(sql, params) binds positional `?` placeholders
and returns rows as backend-agnostic DbValue cells (read out with
DbRowExt::try_get). No trait/async_trait machinery — a closed enum keeps
the pool.get().await? / conn.query(...).await? surface simple.
The Postgres arm rewrites `?` to `$1, $2, ...` (matching
convert_param_style), reuses PgValue for parameter binding, and adds a
Rust-native DbValueFromSql decoder (the non-Python counterpart of
PythonPgFromSql). The shared column-type list is factored into
accepts_column_type so the three mappings can't drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce the native Rust connection pool that both the Rust-native
query path and the Python-facing Connection/Cursor shim will draw from,
so we run a single pool rather than two that could exhaust the server's
connection limit between them.
Uses the generic deadpool::managed pool with our own ConnectionManager
(rather than deadpool-postgres) so connection creation reuses the
existing connect/default-host logic and connection-task spawning. The
pooled item is a plain tokio_postgres::Client, used with the standard
async query functions; recycle() drops connections the driver reports
closed. Live tests are gated on SYNAPSE_TEST_POSTGRES_DSN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement DBAPI2's `executemany`, which Synapse uses for batched writes.
The statement is prepared once and run for each parameter set inside the
connection's (lazily opened) transaction, so a failure part-way aborts
the whole batch. The per-set executions are pipelined — their futures
are driven concurrently so tokio_postgres streams the batch onto the
connection in one round-trip rather than one per statement.
Afterwards `rowcount` reports the total rows affected across all
executions (as psycopg2 does) via a new fetchless "command complete"
cursor state; there is no result set to fetch or describe. An empty
parameter sequence is a no-op that leaves rowcount at -1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The column names for a result set were already carried through the
cursor state machine but kept write-only, awaiting a reader. Add the
PEP-249 `Cursor.description` accessor on top of that plumbing: it
returns one 7-tuple per column (only the name populated, which is all
Synapse reads), or `None` when there is no row-returning result set —
before any query, after an error, or for a column-less statement such
as a bare INSERT, matching psycopg2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Synapse's schema setup (`prepare_database.py`) runs `;`-separated SQL
scripts via the engine's `executescript`. `Cursor.execute` can't serve
those: it `prepare`s the query, and Postgres rejects multiple commands in
a prepared statement.
Add `Cursor.executescript`, which runs the whole script on the simple-query
protocol (`batch_execute`), which does allow multiple statements. It takes
no parameters and produces no fetchable rows.
The script runs inside the connection's *current* transaction (opened
lazily like `execute`) and is left open for the caller to commit. It
deliberately does NOT reproduce the commit-any-pending-transaction-first
behaviour of `sqlite3.executescript` (which psycopg2's engine mirrors with a
leading `COMMIT`). That forced commit actually undercuts the atomicity
`prepare_database` sets out to get — it opens a transaction so "upgrades are
either applied completely, or not at all", but the first script's implicit
commit ends it. Running the script within the ongoing transaction instead
lets successive scripts accumulate and be committed once, which is both
simpler and more correct. The only engine-level piece left to layer on top
is the auto-increment placeholder substitution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously every `tokio_postgres` error became a bare `RuntimeError`. Synapse's
transaction driver, though, branches on the *type* of a database error and on
its `pgcode`: `new_transaction` retries `OperationalError`, retries deadlocks it
recognises via `is_deadlock` (which reads `pgcode`) on a `DatabaseError`, and
`simple_upsert` retries `IntegrityError`. With everything collapsed to
`RuntimeError` none of that fired.
Add just the distinctions Synapse acts on, rather than psycopg2's full PEP-249
hierarchy: `Error` -> `DatabaseError` -> {`OperationalError`, `IntegrityError`},
exposed on the `postgres` submodule, each instance tagged with `pgcode` (the
SQLSTATE string, or `None`). A small classifier maps the SQLSTATE class:
constraint violations (`23`) to `IntegrityError`, connection/resource classes
(`08`/`53`/`57`/`58`) to `OperationalError`, everything else (incl. `40*`
deadlocks, which retry via `pgcode`) to `DatabaseError`. Codeless errors are
split with `is_closed()`: a lost connection is operational, any other (a bad
parameter, a failed connect) is a plain `DatabaseError` so it isn't retried.
Errors surfacing while a result stream is drained (the usual case for an
`INSERT` constraint violation) now route through the same mapping, so they carry
the right class and `pgcode` too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the end-to-end test suite for the Rust DBAPI2 backend, driving the real
tokio-postgres client against a live Postgres server. It is skipped unless
the suite is configured for Postgres (SYNAPSE_POSTGRES), the same switch the
rest of the suite uses.
These cover the behaviours that need a real server rather than the
in-memory fakes the Rust unit tests use: connect (good and bad DSN),
run_interaction (return value, arg/kwarg forwarding, commit on success,
rollback on a Python exception, and recovery after both a Python-raised and
a server-rejected statement, including a constraint violation), cursor
reuse across queries, the fetch_one/fetch_all/fetch_next_batch/rowcount
surface (including batching across a 1000-row result set and the
exhausted-vs-no-active-query error distinction), and value round-trips for
every supported type (NULL, bytea with NUL/high bytes, float4 lossiness,
and an out-of-range int bound to a real int4 column).
A separately-guarded case exercises the libpq default-host fixup by
connecting with a DSN that omits the host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tokio_postgres` and libpq disagree on the default host when a DSN gives
no `host=`: tokio-postgres falls back to localhost, whereas libpq (which
`psql` and the rest of Synapse use) applies its configurable compiled-in
default — typically the Unix socket directory — and honours `PGHOST`.
To keep Synapse's existing connection behaviour, `connect()` now runs the
DSN through `fixup_default_host`: if the parsed config has neither a host
nor a hostaddr, it asks libpq what its default would be (via
`PQconnectStart` on an empty conninfo, which applies libpq's defaults/env
without opening a socket) and sets that on the tokio-postgres config.
The libpq call lives in a small hand-written safe wrapper
(`database::postgres::libpq`) over the `pq-sys` crate. pq-sys ships
pre-generated bindings and links the system libpq itself, so — unlike a
bindgen-based binding crate — this needs no libclang to build. The
behaviour is exercised by the Python integration tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the Python-facing `Connection` / `Cursor` pair and the `connect`
factory, implementing enough of the PEP-249 (DBAPI2) shape for Synapse's
needs.
A `Connection` owns a single `tokio_postgres::Client`. The client *moves*
between the connection and an in-flight cursor rather than being shared:
it lives in the connection between interactions, is taken out for the
duration of a cursor, and is handed back when the cursor finishes. That
single-owner baton (an `Option<Client>` slot on each side) makes it
structurally impossible to use the connection mid-transaction or to run
two overlapping transactions on one socket — both become a clean "already
closed" error.
The transaction lifecycle (`cursor` opens with `BEGIN`; `finish`
COMMIT/ROLLBACKs and hands the client back; `CursorGuard` rolls back an
abandoned transaction on drop) is included here. On any
transaction-control error the client is dropped rather than returned,
closing the socket — safer than handing a possibly-broken connection back
to what will become a connection pool.
`connect()` parses a libpq-style DSN, blocks until connected, and spawns
the long-lived connection task onto the shared runtime (the libpq
default-host fixup is a follow-up). The cursor query methods
(`execute` / `fetch_one` / `fetch_all` / `fetch_next_batch` / `rowcount`)
delegate to the `CursorQueryState` machine.
`run_interaction` — the high-level glue that opens a cursor, runs the
callback, and commits/rolls back — follows in the next change, so
`cursor`/`finish`/`CursorGuard` carry a transitional `#[allow(dead_code)]`
until then. Now that the value/helpers/cursor_state modules are consumed
internally, their visibility is tightened from `pub` back to private.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a batched fetch alongside `fetch_one`/`fetch_all`: it blocks for the
first row, then scoops up any further rows that are already buffered
without blocking again, returning them as one batch. This lets Python
iterate large result sets without the per-row overhead of `fetch_one`,
while still not blocking on the whole result set like `fetch_all`.
Exhaustion is reported by an empty batch, and deliberately deferred: a
batch that runs into the end of the stream still returns the rows it has
and leaves the empty-batch report (and the move to `Closed`) to the next
call. The fused stream makes that re-poll safe.
Unit tests use the in-memory fakes from the previous change, plus a new
`SteppedStream` that can report "not ready yet" mid-stream, so the partial
-batch boundary, the deferred empty report, the capacity-is-a-hint
behaviour, interleaving with `fetch_one`, and the mid-batch error path are
all covered without a live database.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model the lifecycle of a cursor's most recent query as an explicit state
machine (Idle / Active / Closed) so that illegal field combinations are
unrepresentable and fetching past the end of a result set is a clean,
specific error rather than the spurious "connection closed" you get from
re-polling a finished stream. The row stream is fused so an
exhausted-but-not-yet-reported stream can sit safely in `Active`.
This change adds `fetch_one`, `fetch_all` and `rowcount` (the batched
`fetch_next_batch` follows separately). On a stream error a cursor resets
to `Idle`; on normal exhaustion it moves to `Closed`, retaining the
PEP-249 rowcount from the command tag.
The state machine is generic over the stream type, defaulting to
`RowStream`, via a small `CursorRowStream` trait that abstracts the three
things the logic needs (the affected-row count, row->PyTuple conversion,
and error rendering). This is what lets the state transitions, exhaustion
handling and error recovery be unit-tested against an in-memory fake
stream, with no live Postgres server.
`cursor_state` is `pub` for now so its not-yet-consumed items don't trip
clippy's dead_code lint; the connection code wires it up and tightens that
later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Postgres driver is async but the Python-facing API will be
synchronous, so we need a way to drive tokio futures to completion from a
sync, GIL-holding method.
Add a process-global, lazily-initialised multi-thread tokio runtime
(`database::runtime`) whose worker threads only drive the connection
tasks: the actual blocking wait happens on the calling Python thread, so a
small fixed pool can't starve itself.
Add the `block_on` / `block_on_result` / `block_on_next` extension traits
(`database::postgres::helpers`) that block on that runtime while releasing
the GIL (`py.detach`), so other Python threads keep running while we wait.
`pg_err_to_py` maps a driver error into a Python `RuntimeError`.
`BlockingPostgresStream` is implemented generically over `Pin<&mut Fuse<S>>`
rather than just `RowStream`. This is deliberate: it lets the cursor state
machine added in the next change be unit-tested against an in-memory fake
stream, with no live database. The module doc spells out why polling a
`RowStream` off the runtime (the non-blocking fast path) is sound, and why
the `Fuse` bound is required rather than merely assumed.
These modules are `pub` for now so the not-yet-consumed items don't trip
clippy's `dead_code` lint; later changes tighten that.
Unit tests cover the ready / pending / exhausted stream paths (using an
in-memory stream and a `yield_now`-based pending future) and the runtime's
shared-instance behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce the Python<->Postgres value-mapping layer that the forthcoming
Rust DBAPI2 backend will build on, plus the module scaffolding to hang it
off (`synapse_rust.database.postgres`).
`PgValue` is an owned representation of a bound parameter that implements
`tokio_postgres::types::ToSql`, encoding into Postgres' binary wire format
according to the column type from the prepared statement (so a single
Python `int` becomes INT2/INT4/INT8 as appropriate, with range checks).
`PythonPgFromSql` is the decode counterpart, turning a column's wire bytes
back into the natural Python object (and SQL NULL into `None`). The two
`accepts` lists are kept in sync via tests.
The `value` submodule is exposed as `pub` for now so its
not-yet-consumed public items don't trip clippy's `dead_code` lint; later
changes that wire it into the cursor/connection code tighten that back up.
Covers int / float / bool / str / bytes / None; lists and richer types
(json, decimal, timestamps) are left to a follow-up.
The unit tests exercise the whole mapping without a live Postgres server,
including the float4 lossy narrowing, integer width boundaries, the
WrongType and out-of-range paths, and the unsupported/non-UTF-8 decode
errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Impose limit of scheduled delayed events
- Update error codes to match latest draft of MSC4140
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Sync workers proxied a full `ReplicationPresenceSetState` call to the
presence writer on every sync request with `affect_presence=True` (via
`user_syncing`), and a `ReplicationBumpPresenceActiveTime` call on every
user action, even though the writer's presence timers only need feeding
every `SYNC_ONLINE_TIMEOUT` / `LAST_ACTIVE_GRANULARITY`. On busy clients
this amounts to tens of no-op replication calls per user per minute, and
the resulting per-update work is the dominant CPU cost on saturated
presence writers.
Track the last relayed `(state, timestamp)` per `(user, device)` on the
worker and suppress unchanged sync-driven repeats within a 25s relay
interval - deliberately below `SYNC_ONLINE_TIMEOUT` (30s) so the
writer's device `last_sync_ts`/`last_active_ts` timers stay fed and
users neither flap offline nor bounce `currently_active`. Genuine state
changes are relayed immediately, explicit (non-sync) set_state calls
always go through and reset the throttle, bumps that might un-idle a
device bypass it, and entries are evicted when a `USER_SYNC` stop is
sent so reconnecting devices are relayed afresh.
This gives the writer-CPU benefit that deployments currently obtain by
tightening `rc_presence` to ~1/29s, without dropping real presence
transitions.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Quentin Gliech <quenting@element.io>
A handful of places in the storage layer bound a value whose Python type
didn't match the declared column type — an `int` into a `TEXT` column,
or a `str` into a `BIGINT` column — and relied on psycopg2's loose
coercion to paper over the mismatch. These are latent correctness bugs:
they only work because the driver silently converts, and a stricter
driver that binds typed parameters rejects them outright.
Found during a Rust port of the database pool where the driver does not
coerce automatically.
Each fix binds the value with the type the column actually declares,
rather than depending on driver-specific coercion. All changes are
behaviour-preserving on psycopg2 and sqlite.
There is also a fix to the multi-writer id-gen tests where we forgot to
commit. This is tangential, but was found during the same effort.
### Changes
- **`device_lists_remote_extremeties.stream_id (TEXT)`** —
`_update_remote_device_list_cache_txn` (typed `stream_id: int`) bound an
int; store it as a string, matching the column and the sibling
`_update_remote_device_list_cache_entry_txn` (typed `stream_id: str`).
The old mismatch, when rejected, was swallowed inside the device-list
resync and hung `query_devices`.
- **`user_filters.filter_id (BIGINT)`** — `get_user_filter` bound the
raw `int | str` (a string, from sync requests). It already validates via
`int(filter_id)`; bind that int so it matches the column.
- **`rejections.last_check (TEXT)`** — both writers stored
`clock.time_msec()` (an int); store the timestamp as a string.
- **user-directory temp position** — the `populate_user_directory`
background update's staging column
`_temp_populate_user_directory_position.position` was `TEXT` but held an
int (read back into a `BIGINT` column). Declare it `BIGINT`. The temp
table is created and dropped within the background update, so there's no
migration.
- **`test_batched_state_group_storing`** — selected from
`state_group_edges` with a stringified `state_group`; bind the int
directly (the column is an integer).
- **Multi-writer id-generator tests** — constructing a
`MultiWriterIdGenerator` prunes stale `stream_positions` rows, but the
harness never committed, so the cleanup only survived because adbapi
keeps one connection per thread with its transaction open. Commit after
construction so it persists regardless of pool semantics (the delete is
legitimate work that should be committed anyway).
This speeds up the cascading delete from
`sliding_sync_connection_positions`, which without an index on
`connection_position` requires a sequential scan of the whole table for
each deleted position.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Adds `last_active_granularity`, `sync_online_timeout` and `idle_timeout`
options to the `presence` config section, controlling the previously
hard-coded `LAST_ACTIVE_GRANULARITY`, `SYNC_ONLINE_TIMEOUT` and
`IDLE_TIMER` constants (which remain as the defaults).
This is mainly useful on deployments that ratelimit how often syncs can
affect presence (`rc_presence`): the sync timeout must exceed the
ratelimit interval or users flap offline between syncs, so tuning down
presence traffic currently requires squeezing under the fixed 30s limit.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fix https://github.com/element-hq/synapse/issues/19907
The flake manifested as a failure pointing at
`TestMessagesOverFederation/Backfill_from_nearby_backward_extremities_past_token`
but was actually caused by some cross-test pollution from an earlier
test (`TestOIDCProviderUnavailable`) causing some workers to be
temporarily unavailable.
As explained in
https://github.com/element-hq/synapse/issues/19907#issuecomment-4917362461,
> ### Cross-test pollution
>
> When I point an LLM at the logs, it points to
`TestOIDCProviderUnavailable` being the culprit because of the server
restarts polluting this test. When this flake happens, we can indeed see
that `TestOIDCProviderUnavailable` runs before
`TestMessagesOverFederation`.
>
> ```
> PASS TestEventBetweenMakeJoinAndSendJoinIsNotLost 15.08s
> PASS TestFederation/parallel/HS2_->_HS1 1.5s
> PASS TestFederation/parallel/HS1_->_HS2 1.51s
> PASS TestFederation/parallel 0s
> PASS TestFederation 15.12s
> PASS TestOIDCProviderUnavailable//login/sso/redirect_shows_HTML_error
0.02s
> PASS TestOIDCProviderUnavailable 8.62s
> FAIL
TestMessagesOverFederation/Backfill_from_nearby_backward_extremities_past_token
0.89s
> FAIL TestMessagesOverFederation 1.1s
> PASS TestSynapseVersion/Synapse_version_matches_current_git_checkout
0.97s
> PASS TestSynapseVersion 0.97s
> ```
>
> The pollution happens because we enable
[`COMPLEMENT_ENABLE_DIRTY_RUNS`](https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/scripts-dev/complement.sh#L309-L311)
([docs](https://github.com/matrix-org/complement/blob/0e6f8552ff0c99fddb97222399efed3e1f0cb91a/ENVIRONMENT.md#complement_enable_dirty_runs))
which means Complement will reuse deployments (shares homeservers
between tests).
>
> During the `TestOIDCProviderUnavailable` test, there are some stray
federation requests that hit the homeserver while it's still booting
which marks the nginx upstream as unavailable for 10 seconds. nginx has
a default of
[`max_fails=1`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails)
and
[`fail_timeout=10s`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout).
Then when `TestMessagesOverFederation` starts, we're still in the 10
second unavailable window and nginx doesn't even try to connect at all.
>
> <details>
> <summary>LLM summary of the logs and how this happens in
practice</summary>
>
> 1. **19:52:10–11** — the previous federation test finishes:
`user-3:hs2` does a faster-join (`send_join?omit_members=true`) to
`!YnyCRpCLIpimppIreR:hs1`, so `hs2` kicks off a
`sync_partial_state_room` background resync that is still running when
the test ends.
> 2. **19:52:11.6** — `TestOIDCProviderUnavailable` starts and calls
`deployment.StopServer(t, "hs1")` / `StartServer`
(`complement/tests/oidc_test.go:78-80`) to apply an OIDC config
fragment. `hs1` gets `SIGTERM`; new supervisord at 19:52:13.3. `hs2` is
not restarted and keeps retrying its unfinished work.
> 3. **19:52:14** — `hs1`'s nginx is up, but the Synapse workers aren't:
`federation_inbound` only listens on `18015` at 19:52:18.7,
`federation_reader` on `18016` at 19:52:19.0.
> 4. **19:52:15–16** — `hs2`'s retries arrive in that gap: `PUT
/_matrix/federation/v1/send/…` → `18015` refused; `GET
/state_ids/!YnyCRpCLIpimppIreR:hs1` → `18016` refused. With nginx
defaults (`max_fails=1`, `fail_timeout=10s`) and only one server per
upstream block, both upstreams are now marked down until ~19:52:25/26.
(Side casualty: `hs2`'s partial-state resync gives up — "We can't get
valid state history" — and puts `hs1` on a 10-minute federation
backoff.)
> 5. **19:52:21.5** — the failing test's `make_join` for
`@user-5-bob:hs2` reaches `hs1`'s nginx → `no live upstreams` → `502` →
the join fails, even though the worker has been listening for 2.5
seconds by then.
> 6. **19:52:22** — `TestSynapseVersion`'s `GET
/_matrix/federation/v1/version` hits the same dead upstream → `502` → it
fails too.
>
> So it's a flake caused by a race between the OIDC test's container
restart, hs2's background federation retries, and nginx's passive
health-check — not anything wrong with the backfill logic under test.
>
> </details>
>
>
> We can indeed confirm this suspicion with these logs
>
> ❌https://github.com/element-hq/synapse/actions/runs/28888070856/job/85701936736
(archive:
[85701936736.log](https://github.com/user-attachments/files/29809986/85701936736.log)):
> ```
> Error: 2026/07/07 19:52:20 [error] 34#34: *1 no live upstreams while
connecting to upstream, client: 172.18.0.3, server: localhost, request:
"PUT /_matrix/federation/v1/send/1783453927718 HTTP/1.1", upstream:
"http://federation_inbound/_matrix/federation/v1/send/1783453927718",
host: "hs1"
> ...
>
> Error: 2026/07/07 19:52:21 [error] 33#33: *4 no live upstreams while
connecting to upstream, client: 172.18.0.3, server: localhost, request:
"GET
/_matrix/federation/v1/make_join/%21scAbyTQDdaauGYicpU%3Ahs1/%40user-5-bob%3Ahs2?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11
HTTP/1.1", upstream:
"http://federation_reader/_matrix/federation/v1/make_join/%21scAbyTQDdaauGYicpU%3Ahs1/%40user-5-bob%3Ahs2?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11",
host: "hs1"
> ```
>
> The fix here would be to disable nginx's unavailable upstream behavior
by configuring `max_fails=0` in the upstream block:
https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/docker/configure_workers_and_start.py#L374-L378
`test_lock_contention` is a performance-regression canary (#16840): the
pathological behaviour it guards against spent ~30s spinning the CPU, vs
~0.5s when healthy. The 5s wall-clock alarm it used was calibrated on
SQLite, but against PostgreSQL a healthy run already takes 3-4s of
wall-clock time (500 sequential acquire/release cycles, each a real
database round-trip), so any CI load pushed it over the limit.
Add a `cpu_time` mode to `tests/utils.py`'s test_timeout, implemented
with
[`setitimer(ITIMER_PROF)`](https://docs.python.org/3/library/signal.html#signal.setitimer),
which budgets process CPU time instead of wall-clock time. Time spent
blocked on the database or lost to a loaded CI runner no longer counts,
while a regression to CPU-spinning still trips the alarm mid-spin. A
healthy run costs <1s of CPU on either database engine; the budget is
10s.
This also subsumes the RISC-V wall-clock carve-out from #18430, which is
removed.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This is a stepping stone before we can go full Rust everywhere. We're
providing a generic interface as we want database access to work in
Synapse and `synapse-rust-apps`. In `synapse-rust-apps`, we will use a
`tokio-postgres` based database connection pool so it's full Rust.
We want to avoid the situation where we have two database connection
pools (one for Python, one for Rust) as we've run into connection
exhaustion problems on Matrix.org before.
As an example of using it and sanity check for all this work (including
tests), I've also ported over the `/versions` handler to the Rust side
with database access. The `/versions` endpoint is the simplest endpoint
I could find that still had some database access. Hopefully the refactor
on `/versions` isn't that controversial as it's not really the point of
this PR. We can always remove it from this PR but it's just here as a
sanity check that all of this works.
### Why `runInteraction(...)`?
Using the same `runInteraction` pattern that we already have in Synapse
means we can port over existing Synapse code/endpoints without much
thought. But this pattern also makes sense because we want[^1]
transactions to have repeatable-read isolation (easy to think about,
less foot-guns). Having everything thappen in a function callback means
we can do retries for serialization/deadlock errors.
[^1]: To note: Ideally, we'd want the least isolation possible but the
problem is that there is no tooling to yell at you when your
queries/logic is wrong so repeatable-read isolation is a great balance.
> When an application receives this error message, it should abort the
current transaction and retry the whole transaction from the beginning.
The second time through, the transaction will see the
previously-committed change as part of its initial view of the database,
so there is no logical conflict in using the new version of the row as
the starting point for the new transaction's update.
>
> Note that only updating transactions might need to be retried;
read-only transactions will never have serialization conflicts.
>
> *--
https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ*
As a note, this strategy is less of an impedance mismatch (aligns more
closely) with Synapse so the glue code for the `python_db_pool` should
also be simpler.
### How does this interact with logcontext (`LoggingContext`)?
See [docs on log
contexts](https://github.com/element-hq/synapse/blob/4e9f7757f17ba81b8747b7f8f9646d17df145aa3/docs/log_contexts.md)
for more background.
We already support normal logging from Rust -> Python with `pyo3-log`
and `log` but as soon as we pass a thread boundary, everything is logged
against the `sentinel` log context. Normally, we want logs and CPU/DB
usage correlated with the request that spawned the work.
You can see how I took a stab at fixing this in
https://github.com/element-hq/synapse/pull/19846 by capturing the
logcontext in a Tokio task local and re-activating as necessary. For
example, in that PR, I reactivated the logcontext in
`run_python_awaitable(...)` which we use to call `runInteraction(...)`
from the Rust side which means all of the database usage is correlated
with the request as expected. It also means any `log:info!(...)` done in
`run_interaction(...)` is correlated correctly. But there needs to be a
better story for when you want to log everywhere else.
I haven't explored tracking CPU usage on the Rust side.
I've left all of this out of this PR as I think it will be better to
tackle this as a dedicated follow-up. For example, I'm thinking about
instead creating a new `LoggingContext` with the `parent_context` set to
the calling context and try to avoid needing to call
`set_current_context(...)` on the Python side where possible (like
tracking CPU).
### Testing strategy
Added some tests that exercise some `async` Rust handlers for the
`/versions` endpoint:
```
SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_versions.VersionsTestCase
```
Real-world:
1. `poetry run synapse_homeserver --config-path homeserver.yaml`
1. `GET http://localhost:8008/_matrix/client/versions`
The port of event serialization to Rust (#19837) removed
`format_event_raw`, `format_event_for_client_v1`,
`format_event_for_client_v2` and
`format_event_for_client_v2_without_room_id` from
`synapse.events.utils`, but there are modules in the wild that import
them from there.
Reimplement them as standalone pyfunctions in Rust, operating directly
on the Python dict so the original semantics are preserved exactly
(in-place mutation, returning the same dict, arbitrary non-JSON values
passing through, KeyError on a missing `unsigned` in the v1 format), and
re-export them from `synapse.events.utils`.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
### Summary
This moves the synchronous core of client event serialization out of
`synapse/events/utils.py` and into Rust
(`rust/src/events/serialize.rs`).
Event serialization is on the hot path for `/sync`, `/messages`, and
most client-facing endpoints. Previously it was a recursive pure-Python
routine (`_serialize_event` / `_inject_bundled_aggregations` /
`only_fields`) that interleaved
CPU-bound formatting with `async` DB/IO. This PR separates those two
concerns and moves the CPU-bound half to Rust:
- **Async prep stays in Python.**
`EventClientSerializer._prepare_serialization` does all DB/IO up front:
batch-fetching redaction events and running the registered module
`unsigned`-callback hooks, for the top-level events *and* every bundled
sub-event (edits and thread latest events, which are themselves
serialized). The admin/MSC4354 config is resolved once via
`_update_config`, rather than re-checked on every recursive call as the
old code did.
- **The synchronous core moves to Rust.** Given an event plus the
pre-fetched `redaction_map`, `unsigned_additions`, and
`bundle_aggregations`, the Rust code produces the client JSON entirely
in Rust — including the v1/v2 format transforms, `only_event_fields`
filtering, redaction handling, and recursive bundled aggregations.
### Details
- The Rust entry point is a single batch function, `serialize_events`,
taking a list of `(event, membership)` pairs. The three lookup maps are
shared across the whole batch, so they're read out of Python and
converted to Rust
structures **once** per batch rather than once per event.
`EventClientSerializer.serialize_event` (singular) is a thin wrapper
that calls it with a one-element list.
- `SerializeEventConfig` is now a Rust `pyclass`, and the old
`event_format` callable is replaced by the `EventFormat` enum (`Raw` /
`ClientV1` / `ClientV2` / `ClientV2WithoutRoomId`). Call sites in
`rest/admin/events.py` and
`rest/client/{notifications,room,sync}.py` are updated to pass the enum.
`make_config_for_admin` and MSC4354 enablement now go through
`SerializeEventConfig.for_admin()` / `with_msc4354()`.
- New accessors on `EventInternalMetadata` (`redacted_by`, `txn_id`,
`device_id`, `token_id`, `delay_id`, `soft_failed`,
`policy_server_spammy`) expose to Rust the fields the serializer reads.
- The `_split_field` unit tests move from `tests/events/test_utils.py`
to a Rust test in `serialize.rs`, since the implementation moved.
### Behaviour
This is intended to be a behaviour-preserving refactor — the Rust core
mirrors the previous Python output (field ordering, v1 key promotion,
redaction placement per room version, null-`redacts` handling,
transaction-ID gating).
Existing serialization, relations, and sync tests pass unchanged.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Eric Eastwood <erice@element.io>
Speeds up finding and deleting expired sliding sync connections in
`delete_old_sliding_sync_connections`, which previously required a
sequential scan.
On matrix.org I have a suspicion that this might end up blocking some
SSS connections during the delete, which can take minutes. Specifically,
I think the deletion blocks this delete:
https://github.com/element-hq/synapse/blob/ff19c034d300869e64878a15aed9a97f1cec59e4/synapse/storage/databases/main/sliding_sync.py#L233-L241
We didn't previously have an index because we wanted the postgres HOT
updates, however we also limit the update frequency to once every 5
minutes, so hopefully this is fine.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
