Add the end-to-end test suite for the Rust DBAPI2 backend, driving the real
tokio-postgres client against a live Postgres server. It is skipped unless
the suite is configured for Postgres (SYNAPSE_POSTGRES), the same switch the
rest of the suite uses.
These cover the behaviours that need a real server rather than the
in-memory fakes the Rust unit tests use: connect (good and bad DSN),
run_interaction (return value, arg/kwarg forwarding, commit on success,
rollback on a Python exception, and recovery after both a Python-raised and
a server-rejected statement, including a constraint violation), cursor
reuse across queries, the fetch_one/fetch_all/fetch_next_batch/rowcount
surface (including batching across a 1000-row result set and the
exhausted-vs-no-active-query error distinction), and value round-trips for
every supported type (NULL, bytea with NUL/high bytes, float4 lossiness,
and an out-of-range int bound to a real int4 column).
A separately-guarded case exercises the libpq default-host fixup by
connecting with a DSN that omits the host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tokio_postgres` and libpq disagree on the default host when a DSN gives
no `host=`: tokio-postgres falls back to localhost, whereas libpq (which
`psql` and the rest of Synapse use) applies its configurable compiled-in
default — typically the Unix socket directory — and honours `PGHOST`.
To keep Synapse's existing connection behaviour, `connect()` now runs the
DSN through `fixup_default_host`: if the parsed config has neither a host
nor a hostaddr, it asks libpq what its default would be (via
`PQconnectStart` on an empty conninfo, which applies libpq's defaults/env
without opening a socket) and sets that on the tokio-postgres config.
The libpq call lives in a small hand-written safe wrapper
(`database::postgres::libpq`) over the `pq-sys` crate. pq-sys ships
pre-generated bindings and links the system libpq itself, so — unlike a
bindgen-based binding crate — this needs no libclang to build. The
behaviour is exercised by the Python integration tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the Python-facing `Connection` / `Cursor` pair and the `connect`
factory, implementing enough of the PEP-249 (DBAPI2) shape for Synapse's
needs.
A `Connection` owns a single `tokio_postgres::Client`. The client *moves*
between the connection and an in-flight cursor rather than being shared:
it lives in the connection between interactions, is taken out for the
duration of a cursor, and is handed back when the cursor finishes. That
single-owner baton (an `Option<Client>` slot on each side) makes it
structurally impossible to use the connection mid-transaction or to run
two overlapping transactions on one socket — both become a clean "already
closed" error.
The transaction lifecycle (`cursor` opens with `BEGIN`; `finish`
COMMIT/ROLLBACKs and hands the client back; `CursorGuard` rolls back an
abandoned transaction on drop) is included here. On any
transaction-control error the client is dropped rather than returned,
closing the socket — safer than handing a possibly-broken connection back
to what will become a connection pool.
`connect()` parses a libpq-style DSN, blocks until connected, and spawns
the long-lived connection task onto the shared runtime (the libpq
default-host fixup is a follow-up). The cursor query methods
(`execute` / `fetch_one` / `fetch_all` / `fetch_next_batch` / `rowcount`)
delegate to the `CursorQueryState` machine.
`run_interaction` — the high-level glue that opens a cursor, runs the
callback, and commits/rolls back — follows in the next change, so
`cursor`/`finish`/`CursorGuard` carry a transitional `#[allow(dead_code)]`
until then. Now that the value/helpers/cursor_state modules are consumed
internally, their visibility is tightened from `pub` back to private.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a batched fetch alongside `fetch_one`/`fetch_all`: it blocks for the
first row, then scoops up any further rows that are already buffered
without blocking again, returning them as one batch. This lets Python
iterate large result sets without the per-row overhead of `fetch_one`,
while still not blocking on the whole result set like `fetch_all`.
Exhaustion is reported by an empty batch, and deliberately deferred: a
batch that runs into the end of the stream still returns the rows it has
and leaves the empty-batch report (and the move to `Closed`) to the next
call. The fused stream makes that re-poll safe.
Unit tests use the in-memory fakes from the previous change, plus a new
`SteppedStream` that can report "not ready yet" mid-stream, so the partial
-batch boundary, the deferred empty report, the capacity-is-a-hint
behaviour, interleaving with `fetch_one`, and the mid-batch error path are
all covered without a live database.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model the lifecycle of a cursor's most recent query as an explicit state
machine (Idle / Active / Closed) so that illegal field combinations are
unrepresentable and fetching past the end of a result set is a clean,
specific error rather than the spurious "connection closed" you get from
re-polling a finished stream. The row stream is fused so an
exhausted-but-not-yet-reported stream can sit safely in `Active`.
This change adds `fetch_one`, `fetch_all` and `rowcount` (the batched
`fetch_next_batch` follows separately). On a stream error a cursor resets
to `Idle`; on normal exhaustion it moves to `Closed`, retaining the
PEP-249 rowcount from the command tag.
The state machine is generic over the stream type, defaulting to
`RowStream`, via a small `CursorRowStream` trait that abstracts the three
things the logic needs (the affected-row count, row->PyTuple conversion,
and error rendering). This is what lets the state transitions, exhaustion
handling and error recovery be unit-tested against an in-memory fake
stream, with no live Postgres server.
`cursor_state` is `pub` for now so its not-yet-consumed items don't trip
clippy's dead_code lint; the connection code wires it up and tightens that
later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the `block_on`/`block_on_result`/`block_on_next` helpers the Postgres
backend uses to drive its async `tokio-postgres` futures to completion from
sync, GIL-holding Python methods, releasing the GIL for the wait. They take
a `tokio::runtime::Handle` and block on it from the calling (Python) thread.
Rather than give the DB backend a runtime of its own, they use the
extension's existing shared runtime (`tokio_runtime::PyTokioRuntime`, stored
on the reactor). `start` is made idempotent and a `runtime_handle` accessor
starts it on demand, so a caller that needs a connection before the reactor
is running still gets a handle; once the reactor runs, its
`callWhenRunning(start)` hook is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RPEeXx2fAG67o6u4CnmC8W
Introduce the Python<->Postgres value-mapping layer that the forthcoming
Rust DBAPI2 backend will build on, plus the module scaffolding to hang it
off (`synapse_rust.database.postgres`).
`PgValue` is an owned representation of a bound parameter that implements
`tokio_postgres::types::ToSql`, encoding into Postgres' binary wire format
according to the column type from the prepared statement (so a single
Python `int` becomes INT2/INT4/INT8 as appropriate, with range checks).
`PythonPgFromSql` is the decode counterpart, turning a column's wire bytes
back into the natural Python object (and SQL NULL into `None`). The two
`accepts` lists are kept in sync via tests.
The `value` submodule is exposed as `pub` for now so its
not-yet-consumed public items don't trip clippy's `dead_code` lint; later
changes that wire it into the cursor/connection code tighten that back up.
Covers int / float / bool / str / bytes / None; lists and richer types
(json, decimal, timestamps) are left to a follow-up.
The unit tests exercise the whole mapping without a live Postgres server,
including the float4 lossy narrowing, integer width boundaries, the
WrongType and out-of-range paths, and the unsupported/non-UTF-8 decode
errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduced in: #19473
Noticed in:
https://github.com/element-hq/synapse/pull/19556#discussion_r3505783541
I have not experienced the bug in the real world, it's just something I
noticed by reading.
--
Fix bug in `_prune_device_lists_changes_in_room` when transaction is
retried
The `nonlocal` variable is a footgun as it increments the counter even
though the transaction did not commit yet and may still be retried.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
If presence is disabled after having been enabled, the presence states
in the database (and hence on clients) are frozen at whatever they were
when presence was last enabled: nothing writes to the presence stream
any more and /sync omits the presence section entirely, so clients show
the old presence states forever.
Fix this in two parts:
1. At startup, if presence is disabled but the database still contains
non-offline presence states, the presence writer sends out one final
round of updates marking those users as offline.
2. /sync no longer unconditionally omits presence when presence is
disabled: incremental syncs whose since token is behind the presence
stream still get the straggling updates. As the stream doesn't advance
while presence is disabled, clients catch up once and the check then
short-circuits to a token comparison.
Remote servers already handle this themselves by timing out our users
([`FEDERATION_TIMEOUT`](https://github.com/element-hq/synapse/blob/4d8905a15a417ed0054ec2533d243932d890bbbd/synapse/handlers/presence.py#L194-L198)),
so no federation changes are needed.
Note that this only fixes the issue if presence is fully disabled. If
set to `untracked` we still have the same issue, however since modules
would still write to presence we can't just clobber everything like we
do in this patch.
This does two things, first it adds a config flag to ignore rooms for
the purposes of presence routing.
Secondly, it changes the caching behaviour to try and improve the cache
hit ratio. Previously, the size of the `do_users_share_a_room` cache
(which stores pairs of users) needs to `O(n²)` for the number of online
users, which is infeasible for large servers.
Instead, we call `get_users_in_room` for both the syncing and updated
users. This sounds more expensive, but a) we will already have cached
the syncing user's rooms, and b) we will only calculate the updated
user's rooms once (rather than once per syncing user).
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- Impose limit of scheduled delayed events
- Update error codes to match latest draft of MSC4140
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Sync workers proxied a full `ReplicationPresenceSetState` call to the
presence writer on every sync request with `affect_presence=True` (via
`user_syncing`), and a `ReplicationBumpPresenceActiveTime` call on every
user action, even though the writer's presence timers only need feeding
every `SYNC_ONLINE_TIMEOUT` / `LAST_ACTIVE_GRANULARITY`. On busy clients
this amounts to tens of no-op replication calls per user per minute, and
the resulting per-update work is the dominant CPU cost on saturated
presence writers.
Track the last relayed `(state, timestamp)` per `(user, device)` on the
worker and suppress unchanged sync-driven repeats within a 25s relay
interval - deliberately below `SYNC_ONLINE_TIMEOUT` (30s) so the
writer's device `last_sync_ts`/`last_active_ts` timers stay fed and
users neither flap offline nor bounce `currently_active`. Genuine state
changes are relayed immediately, explicit (non-sync) set_state calls
always go through and reset the throttle, bumps that might un-idle a
device bypass it, and entries are evicted when a `USER_SYNC` stop is
sent so reconnecting devices are relayed afresh.
This gives the writer-CPU benefit that deployments currently obtain by
tightening `rc_presence` to ~1/29s, without dropping real presence
transitions.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Quentin Gliech <quenting@element.io>
A handful of places in the storage layer bound a value whose Python type
didn't match the declared column type — an `int` into a `TEXT` column,
or a `str` into a `BIGINT` column — and relied on psycopg2's loose
coercion to paper over the mismatch. These are latent correctness bugs:
they only work because the driver silently converts, and a stricter
driver that binds typed parameters rejects them outright.
Found during a Rust port of the database pool where the driver does not
coerce automatically.
Each fix binds the value with the type the column actually declares,
rather than depending on driver-specific coercion. All changes are
behaviour-preserving on psycopg2 and sqlite.
There is also a fix to the multi-writer id-gen tests where we forgot to
commit. This is tangential, but was found during the same effort.
### Changes
- **`device_lists_remote_extremeties.stream_id (TEXT)`** —
`_update_remote_device_list_cache_txn` (typed `stream_id: int`) bound an
int; store it as a string, matching the column and the sibling
`_update_remote_device_list_cache_entry_txn` (typed `stream_id: str`).
The old mismatch, when rejected, was swallowed inside the device-list
resync and hung `query_devices`.
- **`user_filters.filter_id (BIGINT)`** — `get_user_filter` bound the
raw `int | str` (a string, from sync requests). It already validates via
`int(filter_id)`; bind that int so it matches the column.
- **`rejections.last_check (TEXT)`** — both writers stored
`clock.time_msec()` (an int); store the timestamp as a string.
- **user-directory temp position** — the `populate_user_directory`
background update's staging column
`_temp_populate_user_directory_position.position` was `TEXT` but held an
int (read back into a `BIGINT` column). Declare it `BIGINT`. The temp
table is created and dropped within the background update, so there's no
migration.
- **`test_batched_state_group_storing`** — selected from
`state_group_edges` with a stringified `state_group`; bind the int
directly (the column is an integer).
- **Multi-writer id-generator tests** — constructing a
`MultiWriterIdGenerator` prunes stale `stream_positions` rows, but the
harness never committed, so the cleanup only survived because adbapi
keeps one connection per thread with its transaction open. Commit after
construction so it persists regardless of pool semantics (the delete is
legitimate work that should be committed anyway).
