Rust spans will go to the same Jaeger agent as Python's, over UDP, as
the `Agent.emitBatch` oneway message in the thrift compact protocol.
This adds a write-only encoder for that message and the six structs in
`jaeger.thrift`, with no new dependency.
Spans are encoded one at a time so a reporter knows each span's size
before it joins them into a batch, and `emit_batch_len` bounds a
batch's size without building it.
The output matches jaeger-client's own reporter byte for byte. The
golden fixture is generated by
`scripts-dev/generate_jaeger_thrift_fixture.py` with jaeger-client's
generated thrift code and covers every tag type, logs, a follows-from
reference, a 128-bit trace ID, IDs at or above 2^63 (which thrift
sends negative) and long list headers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
`scripts-dev/pg_roundtrip_proxy.py` sits between Synapse and Postgres. It
counts round trips, a client message after a server message on the same
connection, can add latency to each, and records the protocol messages
and SQL of each round trip. SIGUSR2 resets it and SIGUSR1 writes the
counts and log to a file.
`createroom_bench.py --proxy-delay-ms N` runs the server through it and
reports the round trips per request beside the timings. With a delay,
the extra wall time over a run without one shows how many of those round
trips the request waits for in turn. A new `e2ee_invite` scenario is the
room the E2EE monitor creates on jki.re, and `--scenarios` picks a
subset. `persist_bench.Server` honours `PGPORT`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Each connection of the native Postgres pool caches its prepared
statements. A miss costs a round trip to prepare, and the cache is
emptied whole when it reaches 512 statements. Nothing showed how often
either happens, or how often `txn_limit` replaces a connection and loses
its cache.
The pool now counts cache hits, prepares, cache clears, and connections
opened and retired. Python reports them per pool as
`synapse_rust_db_pool_statement_cache_hits_total`,
`..._statement_prepares_total`, `..._statement_cache_clears_total`,
`..._connections_opened_total` and `..._connections_retired_total`,
carried over when the native pool is closed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`createroom_bench.py` takes levels such as `full+rooms`, which add
`rust_room_creation: on` to a `rust_event_persistence` level, and runs
`off`, `full` and `full+rooms` by default. `persist_bench.Server` takes
extra experimental features for that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`tests/rest/client/bench_createroom.py` times room creation per room in
the trial harness, with optional cProfile dumps and per-room counts of
database interactions and SQL statements. It skips unless
`SYNAPSE_BENCH_CREATEROOM` is set.
`scripts-dev/createroom_bench.py` times it against a real server, reusing
`persist_bench.Server`. That server now lifts the room-creation rate limit
and can run on psycopg2 instead of the native driver.
The test template database's maintenance connection now always uses
psycopg2, as `setup_test_homeserver` does. The Rust driver's `postgres`
module has no `connect`, so it failed under `SYNAPSE_TEST_RUST_DRIVER=1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Brings Rust sliding sync (erikj/rust-sss-queries) together with Rust
event persistence (erikj/rust-persist-validate). Both branch from
a78b02b55c.
The conflicts were both sides adding parallel entries:
- the Complement and test environment switches
- the `RustHandlers` fields and getters
- the `_BlockingRustCalls` types
These keep both. `RustHandlers` builds the persister and the sliding sync
handler in one pass over the native pool. `SyncReader::begin` keeps the
persist side's flag-before-BEGIN and the sliding sync side's transaction
count and `begin_write`. Both sides had added the `invite_room_state`
and `knock_room_state` accessors to `Unsigned`, so one copy is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`scripts-dev/persist_bench.py` starts a real Synapse process per
`rust_event_persistence` level, on a fresh Postgres database with the
native driver, and runs the same workloads through the client API:
concurrent messages, membership churn, joins into a room with many
state events, and a redaction storm. For each, it reports wall time,
the CPU of the reactor thread and of the process, the change in
resident memory, and the `persist_events` transactions from the metrics
endpoint.
On this sandbox, 20 users, two runs of each level:
workload level ms/event txn ms proc CPU s
messages off 19.7 56.7 29.1
messages full 14.4 4.2 23.6
membership off 49.6 7.4 9.5
membership full 41.3 4.4 8.0
big_state_joins off 58.1 7.7 1.05
big_state_joins full 50.5 4.7 0.93
redactions off 13.1 38.5 3.2
redactions full 11.5 7.1 3.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
Phase 0 of the event persistence plan, apart from the write transaction
helper:
- `experimental_features.rust_event_persistence`, one ordered level
(off, shadow, prepare, transaction, full). Nothing uses the levels yet
beyond building the persister.
- `RustHandlers.persister`: built on the events writer when the level is not
off and the main database uses the native driver. It holds the pool, the
shared caches and the events, backfill and cache stream trackers.
- `PersistEffects`, the Python cache updates a Rust transaction will ask for
in place of `txn.call_after`, applied in one call by
`DataStore._apply_rust_persist_effects`. A test persists batches through
Python and checks that applying the matching effects updates the same
caches. The chunk's stream IDs come back unfinished and are finished last,
as Python's `get_next_mult` does after the transaction's callbacks, so
the current token never passes an event the stream change caches don't
know about.
- `PersistContext`, read from an `EventContext` once per event.
- `Event::to_persistence_value` and `EventInternalMetadata::to_persistence_value`:
the `event_json` columns built without the GIL.
- Rust-side persistence counters, added to the Python metrics by a pre-collect
hook. A homeserver can now register more than one such hook. The persisted
position is reported only when it moved, so it can't pull back the gauge
the Python path sets.
- `SYNAPSE_TEST_RUST_PERSIST` for trial and `RUST_PERSIST` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
Phase 0 of the sliding sync port (renders/sliding-sync-rust-plan.md). No
behaviour change: nothing calls the Rust handler yet.
- `experimental_features.rust_sliding_sync: off | on`. There is no shadow
mode. Booleans are accepted, as for `rust_sync`.
- `synapse_rust.sliding_sync.SlidingSyncHandler`, built by `RustHandlers`
on the native driver only. It shares the v2 sync handler's `SyncStore`
and carries the pool, deferred and blocking helpers the later phases'
entry points use.
- `SlidingSyncStats`, observed per request into the
`synapse_sliding_sync_rust_*` histograms. `SyncDbStats` now counts the
transactions a reader begins.
- The `synapse_sliding_sync_rust_rooms_total` and
`synapse_sliding_sync_rust_requests_total` counters.
- `SYNAPSE_TEST_RUST_SLIDING_SYNC` and `_STRICT` for trial, with the handler
wrapped in `_BlockingRustCalls`, and `RUST_SLIDING_SYNC` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The driver's Rust tests are gated on a live server and CI's bare
`cargo test` skipped them all, and nothing ran trial or Complement
through the driver. Give the cargo-test job a Postgres service, add a
trial matrix cell with `SYNAPSE_TEST_RUST_DRIVER=1`, and a Complement
cell that passes `RUST_DRIVER=1` through `complement.sh` to the image's
`SYNAPSE_USE_RUST_DRIVER`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
We haven't done a full schema for about 20 schema versions, so no
surprise!
As a result, I want to produce a full schema soon.
In this PR I dust off `make_full_schema.sh` (which seems to have broken
after some SQLite changes)
and add a CI workflow that runs it (producing a diff) when someone
changes the schema.
The CI workflow also adds a sticky comment showing the diff on the
schema, so you can better appreciate the final effect of a change.
---
**Dead changes:**
I wanted to make it possible to generate a versioned full schema without
the manual work,
but you can't run the background updates without essentially starting up
a homeserver,
at which point it might fail because you haven't run all the deltas yet.
There's no actual good way to do this, short of deleting the latest
deltas (+ tweaking code to not crash without them)
or rolling back in the git history.
Backed out those changes, but they're preserved on the PR if interesting.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Broken out of #20027 because I'd like to have it land first.
> It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
>
> We haven't done a full schema for about 20 schema versions, so no
surprise!
>
> As a result, I want to produce a full schema soon.
When running `make_full_schema.sh`, these drop statements now cause the
error:
```
Parse error near line 2: table event_search_content may not be dropped
Parse error near line 3: table event_search_segments may not be dropped
Parse error near line 4: table event_search_segdir may not be dropped
Parse error near line 5: table event_search_docsize may not be dropped
Parse error near line 6: table event_search_stat may not be dropped
Parse error near line 7: table user_directory_search_content may not be dropped
Parse error near line 8: table user_directory_search_segments may not be dropped
Parse error near line 9: table user_directory_search_segdir may not be dropped
Parse error near line 10: table user_directory_search_docsize may not be dropped
Parse error near line 11: table user_directory_search_stat may not be dropped
```
It seems SQLite has cracked down on code that edits the internal tables.
Because SQLite dumps the schema with `CREATE TABLE IF NOT EXISTS` for
these virtual tables, it's harmless to leave them in the schema dump.
---------
Signed-off-by: Olivier 'reivilibre' <oliverw@matrix.org>
Without this change, it was linking to `develop` as we were on the
`develop` branch because of the previous `merge-back` step at this point
in the release process.
Follow-up to https://github.com/element-hq/synapse/pull/19984 as this
was an oversight and I assumed we would still be on the `release-v1.158`
branch by that point.
This means we can point to a single source of truth instead of
duplicating the content to the tag and GitHub release. Less to manage
and worry about when you make some updates to the changelog (maintenance
burden). We also get to avoid the content sitting in the vendor lock-in
GitHub releases.
We point to
`https://github.com/element-hq/synapse/blob/{repo.active_branch.name}/CHANGES.md`
as it will have the relevant changelog entry at the top and won't change
as we archive releases on `develop`. Even for RC releases after the main
release goes out, the entry will still be towards the top. We could try
to get the heading anchor for the specific section but I thought that it
wasn't necessary (nice but more complex).
When building the v1.155.0rc1, the release notification said the
workflow was successful
but it had actually failed.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This way we actually detect problems like
https://github.com/element-hq/synapse/pull/19475 as they happen instead
of being invisible until something breaks.
Sanity check that Complement is testing against your code changes
(whether it be local or from the PR in CI).
```
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo -run TestSynapseVersion
```
Fixes#19375
`prometheus_client` 0.24 makes `Collector` a generic type.
Previously, `InFlightGauge` inherited from both `Generic[MetricsEntry]`
and `Collector`, resulting in the error `TypeError: cannot create a
consistent MRO` when using `prometheus_client` >= 0.24. This behaviour
of disallowing multiple `Generic` inheritance is more strictly enforced
starting with python 3.14, but can still lead to issues with earlier
versions of python.
This PR separates runtime and typing inheritance for `InFlightGauge`:
- Runtime: `InFlightGauge` inherits only from `Collector`
- Typing: `InFlightGauge` is generic
This preserves static typing, avoids MRO conflicts, and supports both
`prometheus_client` <0.24 and >=0.24.
I have tested these changes out locally with `prometheus_client` 0.23.1
& 0.24 on python 3.14 while sending a bunch of messages over federation
and watching a grafana dashboard configured to show
`synapse_util_metrics_block_in_flight_total` &
`synapse_util_metrics_block_in_flight_real_time_sum` (the only metric
setup to use `InFlightGauge`) and things are working in each case.
https://github.com/element-hq/synapse/blob/a1e9abc7df3e6c43a95cba059348546a4c9d4491/synapse/util/metrics.py#L112-L119
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Store the JSON content of scheduled delayed events as text instead of a
byte array. This brings it in line with the `event_json` table's `json`
column, and fixes the inability to schedule a delayed event with
non-ASCII characters in its content.
Fixes#19242
Fixes https://github.com/element-hq/synapse/issues/19175
This PR moves tracking of what lazy loaded membership we've sent to each
room out of the required state table. This avoids that table from
continuously growing, which massively helps performance as we pull out
all matching rows for the connection when we receive a request.
The new table is only read when we have data in a room to send, so we
end up reading a lot fewer rows from the DB. Though we now read from
that table for every room we have events to return in, rather than once
at the start of the request.
For an explanation of how the new table works, see the
[comment](https://github.com/element-hq/synapse/blob/erikj/sss_better_membership_storage2/synapse/storage/schema/main/delta/93/02_sliding_sync_members.sql#L15-L38)
on the table schema.
The table is designed so that we can later prune old entries if we wish,
but that is not implemented in this PR.
Reviewable commit-by-commit.
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Rather than using dodgy regexes which keep breaking.
Also fixes a regression where it looks like we didn't fail CI if the
delta was in the wrong place.
This is useful as someone downstream can source the
`scripts-dev/complement.sh` script and run the same set of tests as
Synapse:
```bash
# Grab the test packages supported by Synapse.
#
# --fast: Skip rebuilding the docker images,
# --build-only: Will only build Docker images but because we also used `--fast`, it won't do anything.
# `>/dev/null` to redirect stdout to `/dev/null` to get rid of the `echo` logs from the script.
test_packages=$(source ${SYNAPSE_DIR}/scripts-dev/complement.sh --fast --build-only >/dev/null && echo "$SYNAPSE_SUPPORTED_COMPLEMENT_TEST_PACKAGES")
echo $test_packages
```
This is spawning from wanting to run the same set of Complement tests in
the https://github.com/element-hq/synapse-rust-apps project.
This is useful so that the script can be sourced by other scripts
without exiting the calling subshell (composable).
This is split out from https://github.com/element-hq/synapse/pull/19208
to make easy to understand PR's and build up to where we want to go.
The schema lint tries to make sure we don't add or remove indices in
schema files (rather than as background updates), *unless* the table was
created in the same schema file.
The regex to pull out the `CREATE TABLE` SQL incorrectly didn't
recognise `IF NOT EXISTS`.
There is a test delta file that shows that we accept different types of
`CREATE TABLE` and `CREATE INDEX` statements, as well as an index
creation that doesn't have a matching create table (to show that we do
still catch it). The test delta should be removed before merge.
