Two fixes to the `starter` closure that drives a Python awaitable on the
reactor thread on behalf of a tokio task:
- Run the fallible `run_in_background`/`addCallbacks` section inside a
closure so an error can no longer skip the restore: previously the `?`
returned early and permanently leaked the restored context onto the
reactor thread, misattributing all subsequent reactor work.
- Don't restore a context that has already finished (the request completed
or was cancelled while the task was still running — `create_deferred`
does not propagate cancellation). Doing so tripped the 'Re-starting
finished log context' abuse check and accounted work against a context
whose metrics were already finalised; run in the reactor's current
context (normally the sentinel) instead, as the pre-port code did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFbRtswu7rsHrttJFauUUb
`_Sentinel` and its `SENTINEL_CONTEXT` singleton — the root "no logcontext"
marker — are now a native pyclass owned by the Rust logcontext module, which
already owns the "current context" storage. Rust both defines the type and
creates the one instance lazily, so the `register_sentinel` bootstrap (Python
constructing the sentinel and pushing it into the Rust slot to dodge a circular
import) is gone: the module exports `SENTINEL_CONTEXT` directly.
Kept as a separate class from `LoggingContext` (the Null Object pattern, as
upstream has it): the sentinel is falsy, inert (all methods no-op), and
thread-agnostic, the opposite of an active thread-affine context. Merging would
scatter `is_sentinel` branches through `__bool__` and every accounting method.
Invariants preserved: `current_context() is SENTINEL_CONTEXT`, `bool()` is
False, `str()` is "sentinel", and `server_name` is
"unknown_server_from_sentinel_context" (read by the log filter). Verified via
95 trial tests + a Rust≡Python identity/bool/round-trip script; full lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFbRtswu7rsHrttJFauUUb
The operator methods take `other: &ContextResourceUsage`, so passing
`&other` to `add_assign`/`sub_assign` (which already take a reference) is a
double-borrow that fails `cargo clippy -- -D warnings`. Drop the `&`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFbRtswu7rsHrttJFauUUb
Port `set_current_context` — the logcontext switch primitive, on the hottest
path in the logging system — from Python into Rust, and read the thread CPU
usage directly via `getrusage(RUSAGE_THREAD)` (libc) instead of through the
`resource` module. This removes the per-switch Python frame, the `stop`/`start`
method dispatch, and the `struct_rusage` object allocated on every switch;
`current.stop()`/`context.start()` for the common base-`LoggingContext` case
now run inline in Rust.
`usage_start` becomes a private native `(ru_utime, ru_stime)` snapshot rather
than a Python `struct_rusage` — nothing outside the module read it. The
`start`/`stop` methods keep their abuse-detection behaviour (thread affinity,
double-start, stop-without-start) via a shared native core (`start_inner`/
`stop_inner`) that both the pymethods and the switch fast path call, so those
checks still run on every switch. `_get_cputime` folds into a native
`cputime_delta` helper.
Subclasses (`BackgroundProcessLoggingContext`) still work unchanged: the switch
dispatches to them through Python (`obj.start(rusage)` / `obj.stop(rusage)`),
materialising the rusage as a plain `(utime, stime)` tuple only on that
off-hot-path branch, so their overrides run. `getrusage(RUSAGE_THREAD)` is
Linux-only; other platforms return `None` (no per-context CPU accounting), as
before.
Removes the now-dead Python rusage plumbing (`get_thread_resource_usage`, the
`RUSAGE_THREAD` support probe, `is_thread_resource_usage_supported`) — Rust owns
rusage now. The Phase 0 characterization test's rusage stand-in changes from an
opaque `object()` to a `(0.0, 0.0)` tuple to match the native representation;
it still pins the same observable abuse-detection behaviour.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LnJSXQ86AAWNtihR4C5PH
Move `LoggingContext` from a pure-Python class to a native `#[pyclass]` in
`rust/src/logcontext.rs`, re-exported from `synapse.logging.context`. The
attribute surface, methods, error-message wording and abuse-detection
behaviour are all preserved, so callers — and the Python subclass
`BackgroundProcessLoggingContext` — are unaffected.
`set_current_context` stays in Python for now: it still calls `ctx.start`/
`ctx.stop` via normal attribute dispatch, so subclass overrides continue to
work. The native switch primitive (with in-Rust `getrusage`) is a later
increment.
Notes on the port:
- Construction is split between `__new__` (allocates a blank instance) and
`__init__` (does the real work), mirroring a pure-Python class. This is what
lets `BackgroundProcessLoggingContext`, which composes a name and then calls
`super().__init__(name=..., server_name=...)`, keep working unchanged.
- Methods that both mutate `self` and call back into Python (`start`, `stop`,
`__enter__`, `__exit__`) take an owned `Bound<Self>` and only hold a
`borrow_mut()` across the field write itself, never across a Python call, so
a re-entrant log record (which resolves the current context and reads this
object's getters) cannot hit an "already borrowed" error.
- `logcontext_error`, `set_current_context`, `logger` and
`logcontext_debug_logger` are re-resolved through the module on every call so
that test patches take effect.
- `__traverse__`/`__clear__` cover `previous_context`/`parent_context`/
`request`/`scope` for the cyclic GC (scope <-> context is a real cycle).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LnJSXQ86AAWNtihR4C5PH
Replace the Python `ContextResourceUsage` class with a native `#[pyclass]`
(exposed via `synapse.synapse_rust.logcontext`, re-exported under the same name
from `synapse.logging.context`). The public surface is preserved drop-in:
the six get/set attributes, `copy`/`reset`, `+`/`-`/`+=`/`-=`, the
`ContextResourceUsage(copy_from=...)` constructor and the exact `repr`.
This is the first step of moving the logcontext machinery to Rust: keeping the
usage accounting in a native struct lets the switch path (a later commit) do
its rusage bookkeeping without allocating a Python object per operation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LnJSXQ86AAWNtihR4C5PH
Move the "current logcontext" storage out of the Python `threading.local`
and into a Rust slot (`rust/src/logcontext.rs`, exposed as
`synapse.synapse_rust.logcontext`). `current_context()` now consults the
tokio task-local first, then the OS thread-local, then the sentinel.
This makes a single source of truth visible from both worlds:
- Log records emitted from Rust while a tokio task is being polled (e.g.
reqwest connecting, and any `log::` records from our async code or its
dependencies) resolve `current_context()` to the context that was current
when Python called into Rust, so `LoggingContextFilter` attributes them
correctly — no per-record stamping machinery, pyo3-log unchanged.
- `run_python_awaitable` now restores that captured context on the reactor
thread before driving the awaitable, so Python invoked from Rust (e.g.
`DatabasePool.runInteraction` from the native `/versions` handler) runs in
the right logcontext and its DB-transaction accounting lands on the right
request. Resolves the FIXME at deferred.rs:223.
`create_deferred` captures the caller's context at the FFI boundary and
scopes it onto the spawned task. Python keeps the accounting policy:
`set_current_context` still does the `getrusage` start/stop bookkeeping and
only delegates the raw slot write to `swap_current_context`. The sentinel is
pushed into Rust at import (`register_sentinel`) so `context is
SENTINEL_CONTEXT` identity and `bool(context)` semantics are preserved, and
Rust never imports `synapse.logging.context` (no circular import).
Tests: Rust unit tests for the slot/lookup precedence; a trial test that a
Rust-origin log record is attributed to the caller's logcontext; and a trial
test that the `/versions` per-user DB lookup is accounted against the caller's
logcontext (the resolved FIXME). Existing tests/util/test_logcontext.py and
tests/synapse_rust pass unmodified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LnJSXQ86AAWNtihR4C5PH
- Impose limit of scheduled delayed events
- Update error codes to match latest draft of MSC4140
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
This is a stepping stone before we can go full Rust everywhere. We're
providing a generic interface as we want database access to work in
Synapse and `synapse-rust-apps`. In `synapse-rust-apps`, we will use a
`tokio-postgres` based database connection pool so it's full Rust.
We want to avoid the situation where we have two database connection
pools (one for Python, one for Rust) as we've run into connection
exhaustion problems on Matrix.org before.
As an example of using it and sanity check for all this work (including
tests), I've also ported over the `/versions` handler to the Rust side
with database access. The `/versions` endpoint is the simplest endpoint
I could find that still had some database access. Hopefully the refactor
on `/versions` isn't that controversial as it's not really the point of
this PR. We can always remove it from this PR but it's just here as a
sanity check that all of this works.
### Why `runInteraction(...)`?
Using the same `runInteraction` pattern that we already have in Synapse
means we can port over existing Synapse code/endpoints without much
thought. But this pattern also makes sense because we want[^1]
transactions to have repeatable-read isolation (easy to think about,
less foot-guns). Having everything thappen in a function callback means
we can do retries for serialization/deadlock errors.
[^1]: To note: Ideally, we'd want the least isolation possible but the
problem is that there is no tooling to yell at you when your
queries/logic is wrong so repeatable-read isolation is a great balance.
> When an application receives this error message, it should abort the
current transaction and retry the whole transaction from the beginning.
The second time through, the transaction will see the
previously-committed change as part of its initial view of the database,
so there is no logical conflict in using the new version of the row as
the starting point for the new transaction's update.
>
> Note that only updating transactions might need to be retried;
read-only transactions will never have serialization conflicts.
>
> *--
https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ*
As a note, this strategy is less of an impedance mismatch (aligns more
closely) with Synapse so the glue code for the `python_db_pool` should
also be simpler.
### How does this interact with logcontext (`LoggingContext`)?
See [docs on log
contexts](https://github.com/element-hq/synapse/blob/4e9f7757f17ba81b8747b7f8f9646d17df145aa3/docs/log_contexts.md)
for more background.
We already support normal logging from Rust -> Python with `pyo3-log`
and `log` but as soon as we pass a thread boundary, everything is logged
against the `sentinel` log context. Normally, we want logs and CPU/DB
usage correlated with the request that spawned the work.
You can see how I took a stab at fixing this in
https://github.com/element-hq/synapse/pull/19846 by capturing the
logcontext in a Tokio task local and re-activating as necessary. For
example, in that PR, I reactivated the logcontext in
`run_python_awaitable(...)` which we use to call `runInteraction(...)`
from the Rust side which means all of the database usage is correlated
with the request as expected. It also means any `log:info!(...)` done in
`run_interaction(...)` is correlated correctly. But there needs to be a
better story for when you want to log everywhere else.
I haven't explored tracking CPU usage on the Rust side.
I've left all of this out of this PR as I think it will be better to
tackle this as a dedicated follow-up. For example, I'm thinking about
instead creating a new `LoggingContext` with the `parent_context` set to
the calling context and try to avoid needing to call
`set_current_context(...)` on the Python side where possible (like
tracking CPU).
### Testing strategy
Added some tests that exercise some `async` Rust handlers for the
`/versions` endpoint:
```
SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_versions.VersionsTestCase
```
Real-world:
1. `poetry run synapse_homeserver --config-path homeserver.yaml`
1. `GET http://localhost:8008/_matrix/client/versions`
The port of event serialization to Rust (#19837) removed
`format_event_raw`, `format_event_for_client_v1`,
`format_event_for_client_v2` and
`format_event_for_client_v2_without_room_id` from
`synapse.events.utils`, but there are modules in the wild that import
them from there.
Reimplement them as standalone pyfunctions in Rust, operating directly
on the Python dict so the original semantics are preserved exactly
(in-place mutation, returning the same dict, arbitrary non-JSON values
passing through, KeyError on a missing `unsigned` in the v1 format), and
re-export them from `synapse.events.utils`.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
### Summary
This moves the synchronous core of client event serialization out of
`synapse/events/utils.py` and into Rust
(`rust/src/events/serialize.rs`).
Event serialization is on the hot path for `/sync`, `/messages`, and
most client-facing endpoints. Previously it was a recursive pure-Python
routine (`_serialize_event` / `_inject_bundled_aggregations` /
`only_fields`) that interleaved
CPU-bound formatting with `async` DB/IO. This PR separates those two
concerns and moves the CPU-bound half to Rust:
- **Async prep stays in Python.**
`EventClientSerializer._prepare_serialization` does all DB/IO up front:
batch-fetching redaction events and running the registered module
`unsigned`-callback hooks, for the top-level events *and* every bundled
sub-event (edits and thread latest events, which are themselves
serialized). The admin/MSC4354 config is resolved once via
`_update_config`, rather than re-checked on every recursive call as the
old code did.
- **The synchronous core moves to Rust.** Given an event plus the
pre-fetched `redaction_map`, `unsigned_additions`, and
`bundle_aggregations`, the Rust code produces the client JSON entirely
in Rust — including the v1/v2 format transforms, `only_event_fields`
filtering, redaction handling, and recursive bundled aggregations.
### Details
- The Rust entry point is a single batch function, `serialize_events`,
taking a list of `(event, membership)` pairs. The three lookup maps are
shared across the whole batch, so they're read out of Python and
converted to Rust
structures **once** per batch rather than once per event.
`EventClientSerializer.serialize_event` (singular) is a thin wrapper
that calls it with a one-element list.
- `SerializeEventConfig` is now a Rust `pyclass`, and the old
`event_format` callable is replaced by the `EventFormat` enum (`Raw` /
`ClientV1` / `ClientV2` / `ClientV2WithoutRoomId`). Call sites in
`rest/admin/events.py` and
`rest/client/{notifications,room,sync}.py` are updated to pass the enum.
`make_config_for_admin` and MSC4354 enablement now go through
`SerializeEventConfig.for_admin()` / `with_msc4354()`.
- New accessors on `EventInternalMetadata` (`redacted_by`, `txn_id`,
`device_id`, `token_id`, `delay_id`, `soft_failed`,
`policy_server_spammy`) expose to Rust the fields the serializer reads.
- The `_split_field` unit tests move from `tests/events/test_utils.py`
to a Rust test in `serialize.rs`, since the implementation moved.
### Behaviour
This is intended to be a behaviour-preserving refactor — the Rust core
mirrors the previous Python output (field ordering, v1 key promotion,
redaction placement per room version, null-`redacts` handling,
transaction-ID gating).
Existing serialization, relations, and sync tests pass unchanged.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Eric Eastwood <erice@element.io>
This is in prep for converting the event serialization to Rust.
This is a fairly mechanical port, except that we store the appservice ID
rather than the appservice object. This avoids us having to store a
`Py<..>` (or port the appservice object over).
Follow on from #19701
This (more or less) matches what we had before. Otherwise we just get a
default `<builtins.Event at 0x...>`
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Follow on from https://github.com/element-hq/synapse/pull/19701.
Unfortunately serde has a bug when using `#[serde(flatten)]` with
`arbitrary-precision` feature when handling integers that fit in a i128
when doing `serde_json::from_value`. See
https://github.com/serde-rs/serde/issues/2230.
The `depythonize` hits the same issue. To fix this we make it so we only
parse events from strings and not values.
Follow on from #19701.
Some Synapse servers may have events in their database that don't pass
the canonical JSON checks. This is bad, but we still want to be able to
load them nonetheless.
Ports the event class to Rust.
The main difference here are:
1. There is now a single event class
2. We now validate a lot more at event construction time than we
previously did (we basically checked nothing before). This required some
changes to the tests, including
https://github.com/matrix-org/sytest/pull/1423
Reviewable commit-by-commit.
### Overview of Event Rust structure
The format of the event struct in Rust is quite different than that in
Python.
The top-level looks like:
```rust
pub struct Event {
/// The parsed event JSON.
fields: FormattedEvent,
/// The event ID. For format v1 this is read directly from the JSON;
/// for v2+ it is computed from the canonical-JSON hash at
/// construction time and cached here.
event_id: Arc<str>,
/// Synapse-internal per-event state that lives outside the federated
/// JSON (e.g. outlier flag, soft-failure, stream positions).
#[pyo3(get)]
internal_metadata: EventInternalMetadata,
/// The room version this event was parsed for.
#[pyo3(get)]
room_version: &'static RoomVersion,
/// `None` for accepted events; otherwise a short reason set by auth
/// when the event was rejected.
rejected_reason: Option<Box<str>>,
}
```
which includes the actual parsed event in `FormattedEvent`, plus the
rest of the event metadata.
```rust
pub struct FormattedEvent<E = Arc<EventFormatEnum>> {
#[serde(default)]
pub signatures: Signatures,
#[serde(default)]
pub unsigned: Unsigned,
#[serde(flatten)]
pub specific_fields: E,
#[serde(flatten)]
pub common_fields: Arc<EventCommonFields>,
}
```
The struct is further split into the common fields, format specific
fields, plus the signatures and unsigned. We split out the signature and
unsigned fields as they are mutable, so when we clone the event we can
still share the common and specific fields and only copy signature and
unsigned.
The `specific_fields` are the fields that depend on the format version.
They can either be a specific format (e.g. `E = EventFormatV1`) or a
type-erased enum `EventFormatEnum` that is across all room versions:
```rust
pub enum EventFormatEnum {
V1(EventFormatV1),
V2V3(EventFormatV2V3),
V4(EventFormatV4),
VMSC4242(EventFormatVMSC4242),
}
```
For example:
```rust
/// Shared flat-list encoding of `auth_events` and `prev_events`, reused
/// by every format from v2/v3 onwards.
#[derive(Serialize, Deserialize)]
pub struct SimpleAuthPrevEvents {
pub auth_events: Vec<String>,
pub prev_events: Vec<String>,
}
/// Version-specific fields for room versions 3-10.
#[derive(Serialize, Deserialize)]
pub struct EventFormatV2V3 {
pub room_id: Box<str>,
#[serde(flatten)]
pub auth_prev_events: SimpleAuthPrevEvents,
}
```
### Dev notes
As discussed in
[`#element-backend-internal:matrix.org`](https://matrix.to/#/!SGNQGPGUwtcPBUotTL:matrix.org/$3gTjDO440GbAz57cXcCawwiyFLiD0crrarvS1uhzKOY?via=jki.re&via=element.io&via=matrix.org)
---------
Co-authored-by: Eric Eastwood <erice@element.io>
This is in prep for using the room versions more from Rust.
Main changes:
- Change it so each room version is defined as a delta to the last one.
This is a cosmetic change that makes it easier to ensure the room
version definitions are correct (as they're defined as deltas from
previous versions).
- Move constants to `RoomVersion` constants, like `RoomVersion::V1`, for
convenience.
- Change visibility of various attributes.
Based on #19708.
This is on the path to porting the entire event class to Rust, as
`event.content` will then return the new Rust class `JsonObject`.
This PR adds a pure Rust `JsonObject` class that is a `Mapping`
representing a json-style object. It uses `serde_json::Value` as its
in-memory representation and `pythonize` for conversion when a field is
looked up on the object.
I'm not thrilled with the name, but couldn't think of a better one.
This also adds `JsonObject` handling to the JSON serialisation functions
we use, as well as to the `freeze(..)` function.
Reviewable commit-by-commit.
Similar to #19706, let's port the `unsigned` field into a Rust class.
This does change things a bit in that we now define exactly what
unsigned fields that are allowed to be added to an event, and what
actually gets persisted. This should be a noop though, as we carefully
filter out what unsigned fields we allow in from federation, for example
As a side effect of this cleanup, I think this fixes handling
`unsigned.age` on events received over federation.
This is another stepping stone in porting the event class fully to Rust.
The new `Signatures` class is relatively simple, as we actually don't
interact with it that much in the code. It does *not* implement
`Mapping` or `MutableMapping` as that takes quite a lot of effort that
we don't need, even though it would be more ergonomic.
This comes from
https://github.com/erikjohnston/rust-signed-json/blob/main/src/json.rs.
We need to be able to serialise canonical JSON in Rust to be able to
calculate event IDs once we port the event class to Rust.
We could instead make the above a properly published crate, but feels
easier to pull it into Synapse utils.
This moves the reference counting from PyO3 into standard Rust types,
allowing the class to be used natively from Rust without needing a
Python runtime.
Follows: #19365
Part of: MSC4354 Sticky Events (experimental feature #19409)
This PR introduces a `spam_checker_spammy` flag, analogous to
`policy_server_spammy`, as an explicit flag
that an event was decided to be spammy by a spam-checker module.
The original Sticky Events PR (#18968) just reused
`policy_server_spammy`, but it didn't sit right with me
because we (at least appear to be experimenting with features that)
allow users to opt-in to seeing
`policy_server_spammy` events (presumably for moderation purposes).
Keeping these flags separate felt best, therefore.
As for why we need this flag: soon soft-failed status won't be
permanent, at least for sticky events.
The spam checker modules currently work by making events soft-failed.
We want to prevent spammy events from getting
reconsidered/un-soft-failed, so it seems like we need
a flag to track spam-checker spamminess *separately* from soft-failed.
Should be commit-by-commit friendly, but is also small.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
The Rust port of `KNOWN_ROOM_VERSIONS` (#19589) made `__contains__`
strict about key types, raising `TypeError` when called with `None`
instead of returning `False` like a Python dict would.
This broke `/sync` for rooms with a NULL `room_version` in the database.
```
File "/home/synapse/src/synapse/handlers/sync.py", line 2628, in _get_room_changes_for_initial_sync
if event.room_version_id not in KNOWN_ROOM_VERSIONS:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument 'key': 'NoneType' object cannot be cast as 'str'
```
# What is done?
- resolves#19403
- Adds build-time Rust compiler detection and captures the rustc
--version value during the build.
- Exposes the captured compiler version from the Rust extension via a
new Python-callable function.
- Exports a new Prometheus metric for rustc version.
# How to test?
- compile `poetry install`
- add `enable_metrics: true` and
```yaml
resources:
- compress: false
names:
- client
- federation
- metrics
```
to homeserver.yaml
- start synapse
- find the rustc version at `http://localhost:8008/_synapse/metrics`
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
This is a simplification so that `unsigned` only includes "simple"
values, to make it easier to port to Rust.
Reviewable commit-by-commit
Summary:
1. **Add `recheck` column to `redactions` table**
A new boolean `recheck` column (default true) is added to the
`redactions` table. This captures whether a redaction needs its sender
domain checked at read time — required for room v3+ where redactions are
accepted speculatively and later validated. When persisting a new
redaction, `recheck` is set directly from
`event.internal_metadata.need_to_check_redaction()`.
It's fine if initially we recheck all redactions, as it only results in
a little more CPU overhead (as we always pull out the redaction event
regardless).
2. **Backfill `recheck` via background update**
A background update (`redactions_recheck`) backfills the new column for
existing rows by reading `recheck_redaction` from each event's
`internal_metadata` JSON. This avoids loading full event objects by
reading `event_json` directly via a SQL JOIN.
3. **Don't fetch confirmed redaction events from the DB**
Previously, when loading events, Synapse recursively fetched all
redaction events regardless of whether they needed domain rechecking.
Now `_fetch_event_rows` reads the `recheck` column and splits redactions
into two lists:
- `unconfirmed_redactions` — need fetching and domain validation
- `confirmed_redactions` — already validated, applied directly without
fetching the event
This avoids unnecessary DB reads for the common case of
already-confirmed redactions.
4. **Move `redacted_because` population to `EventClientSerializer`**
Previously, `redacted_because` (the full redaction event object) was
stored in `event.unsigned` at DB fetch time, coupling storage-layer code
to client serialization concerns. This is removed from
`_maybe_redact_event_row` and moved into
`EventClientSerializer.serialize_event`, which fetches the redaction
event on demand. The storage layer now only sets
`unsigned["redacted_by"]` (the redaction event ID).
5. **Always use `EventClientSerializer`**
The standalone `serialize_event` function was made private
(`_serialize_event`). All external callers — `rest/client/room.py`,
`rest/admin/events.py, appservice/api.py`, and `tests` — were updated to
use `EventClientSerializer.serialize_event` / `serialize_events`,
ensuring
`redacted_because` is always populated correctly via the serializer.
6. **Batch-fetch redaction events in `serialize_events`**
`serialize_events` now collects all `redacted_by` IDs from the event
batch upfront and fetches them in a single `get_events` call, passing
the result as a `redaction_map` to each `serialize_event` call. This
reduces N individual DB round-trips to one when serializing a batch of
events that includes redacted events.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Principally so that we can share the same room version configuration
between Python and Rust.
For the most part, this is a direct port. Some special handling has had
to go into `KNOWN_ROOM_VERSIONS` so that it can be sensibly shared
between Python and Rust, since we do update it during config parsing.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This allows the Rust HTTP client to be configured to force HTTP/2 even
on plaintext connections. This is useful in contexts where the remote
server is known to server HTTP/2 over plain text.
Added because we use the Synapse Rust HTTP client with the Synapse Pro
`event-cache` module. We use this because it's independent from the
Python reactor which makes things slower than expected.
Currently, the Synapse Rust HTTP client uses HTTP/1 which means a new
connection for every request. With HTTP/2, we can share the connection
across requests.
We want to see if this will make a performance difference and less
stress on the database connection situation, see
https://github.com/element-hq/synapse-rust-apps/issues/452#issuecomment-3897717599
Here is the sibling PR for using HTTP/2 on the Synapse Pro `event-cache`
module side: https://github.com/element-hq/synapse-pro-modules/pull/35
Hello,
I'm writing on behalf of the Citadel product developed by ERCOM.
This PR bumps `pyo3` from 0.26.0 to 0.27.2 and `pythonize` from 0.26.0
to 0.27.0.
For the code migration I followed the guide found here:
[link](https://pyo3.rs/v0.27.0/migration.html).
This changes the arguments in clock functions to be `Duration` and
converts call sites and constants into `Duration`. There are still some
more functions around that should be converted (e.g.
`timeout_deferred`), but we leave that to another PR.
We also changes `.as_secs()` to return a float, as the rounding broke
things subtly. The only reason to keep it (its the same as
`timedelta.total_seconds()`) is for symmetry with `as_millis()`.
Follows on from https://github.com/element-hq/synapse/pull/19223
