Phase 0 of the event persistence plan, apart from the write transaction
helper:
- `experimental_features.rust_event_persistence`, one ordered level
(off, shadow, prepare, transaction, full). Nothing uses the levels yet
beyond building the persister.
- `RustHandlers.persister`: built on the events writer when the level is not
off and the main database uses the native driver. It holds the pool, the
shared caches and the events, backfill and cache stream trackers.
- `PersistEffects`, the Python cache updates a Rust transaction will ask for
in place of `txn.call_after`, applied in one call by
`DataStore._apply_rust_persist_effects`. A test persists batches through
Python and checks that applying the matching effects updates the same
caches. The chunk's stream IDs come back unfinished and are finished last,
as Python's `get_next_mult` does after the transaction's callbacks, so
the current token never passes an event the stream change caches don't
know about.
- `PersistContext`, read from an `EventContext` once per event.
- `Event::to_persistence_value` and `EventInternalMetadata::to_persistence_value`:
the `event_json` columns built without the GIL.
- Rust-side persistence counters, added to the Python metrics by a pre-collect
hook. A homeserver can now register more than one such hook. The persisted
position is reported only when it moved, so it can't pull back the gauge
the Python path sets.
- `SYNAPSE_TEST_RUST_PERSIST` for trial and `RUST_PERSIST` for Complement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHKYPSc37kWqGACn9aKxKL
On the native Postgres driver, the Rust queues' `NativeHooks` read the
device EDUs and record finished transactions through `FederationStore`,
on tokio. A transaction then takes no trip to the reactor, unless it
finishes a sticky event batch or a module wants to hear about delivered
events. Catch-up and the sticky event backlog stay Python. SQLite and
the Python Postgres driver keep the Python hooks.
Python skips the device queries for destinations with nothing new by
asking two stream change caches. Rust keeps a mirror of each, per
destination, fed by a new `StreamChangeCache.add_listener`, so it hears
of every change the caches do. The current stream positions live in
Python's ID generators, and are read under the GIL (without a reactor
trip) only when a destination has changed.
The loops now count their transactions, EDUs and requests in Rust.
`synapse.metrics.pre_collect_hooks` adds the counts to the Python
metrics when they are collected. This removes a GIL crossing on every
failed send, and from `_rust_complete`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The driver's Rust tests are gated on a live server and CI's bare
`cargo test` skipped them all, and nothing ran trial or Complement
through the driver. Give the cargo-test job a Postgres service, add a
trial matrix cell with `SYNAPSE_TEST_RUST_DRIVER=1`, and a Complement
cell that passes `RUST_DRIVER=1` through `complement.sh` to the image's
`SYNAPSE_USE_RUST_DRIVER`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
We haven't done a full schema for about 20 schema versions, so no
surprise!
As a result, I want to produce a full schema soon.
In this PR I dust off `make_full_schema.sh` (which seems to have broken
after some SQLite changes)
and add a CI workflow that runs it (producing a diff) when someone
changes the schema.
The CI workflow also adds a sticky comment showing the diff on the
schema, so you can better appreciate the final effect of a change.
---
**Dead changes:**
I wanted to make it possible to generate a versioned full schema without
the manual work,
but you can't run the background updates without essentially starting up
a homeserver,
at which point it might fail because you haven't run all the deltas yet.
There's no actual good way to do this, short of deleting the latest
deltas (+ tweaking code to not crash without them)
or rolling back in the git history.
Backed out those changes, but they're preserved on the PR if interesting.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Broken out of #20027 because I'd like to have it land first.
> It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
>
> We haven't done a full schema for about 20 schema versions, so no
surprise!
>
> As a result, I want to produce a full schema soon.
When running `make_full_schema.sh`, these drop statements now cause the
error:
```
Parse error near line 2: table event_search_content may not be dropped
Parse error near line 3: table event_search_segments may not be dropped
Parse error near line 4: table event_search_segdir may not be dropped
Parse error near line 5: table event_search_docsize may not be dropped
Parse error near line 6: table event_search_stat may not be dropped
Parse error near line 7: table user_directory_search_content may not be dropped
Parse error near line 8: table user_directory_search_segments may not be dropped
Parse error near line 9: table user_directory_search_segdir may not be dropped
Parse error near line 10: table user_directory_search_docsize may not be dropped
Parse error near line 11: table user_directory_search_stat may not be dropped
```
It seems SQLite has cracked down on code that edits the internal tables.
Because SQLite dumps the schema with `CREATE TABLE IF NOT EXISTS` for
these virtual tables, it's harmless to leave them in the schema dump.
---------
Signed-off-by: Olivier 'reivilibre' <oliverw@matrix.org>
Without this change, it was linking to `develop` as we were on the
`develop` branch because of the previous `merge-back` step at this point
in the release process.
Follow-up to https://github.com/element-hq/synapse/pull/19984 as this
was an oversight and I assumed we would still be on the `release-v1.158`
branch by that point.
This means we can point to a single source of truth instead of
duplicating the content to the tag and GitHub release. Less to manage
and worry about when you make some updates to the changelog (maintenance
burden). We also get to avoid the content sitting in the vendor lock-in
GitHub releases.
We point to
`https://github.com/element-hq/synapse/blob/{repo.active_branch.name}/CHANGES.md`
as it will have the relevant changelog entry at the top and won't change
as we archive releases on `develop`. Even for RC releases after the main
release goes out, the entry will still be towards the top. We could try
to get the heading anchor for the specific section but I thought that it
wasn't necessary (nice but more complex).
When building the v1.155.0rc1, the release notification said the
workflow was successful
but it had actually failed.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This way we actually detect problems like
https://github.com/element-hq/synapse/pull/19475 as they happen instead
of being invisible until something breaks.
Sanity check that Complement is testing against your code changes
(whether it be local or from the PR in CI).
```
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo -run TestSynapseVersion
```
Fixes#19375
`prometheus_client` 0.24 makes `Collector` a generic type.
Previously, `InFlightGauge` inherited from both `Generic[MetricsEntry]`
and `Collector`, resulting in the error `TypeError: cannot create a
consistent MRO` when using `prometheus_client` >= 0.24. This behaviour
of disallowing multiple `Generic` inheritance is more strictly enforced
starting with python 3.14, but can still lead to issues with earlier
versions of python.
This PR separates runtime and typing inheritance for `InFlightGauge`:
- Runtime: `InFlightGauge` inherits only from `Collector`
- Typing: `InFlightGauge` is generic
This preserves static typing, avoids MRO conflicts, and supports both
`prometheus_client` <0.24 and >=0.24.
I have tested these changes out locally with `prometheus_client` 0.23.1
& 0.24 on python 3.14 while sending a bunch of messages over federation
and watching a grafana dashboard configured to show
`synapse_util_metrics_block_in_flight_total` &
`synapse_util_metrics_block_in_flight_real_time_sum` (the only metric
setup to use `InFlightGauge`) and things are working in each case.
https://github.com/element-hq/synapse/blob/a1e9abc7df3e6c43a95cba059348546a4c9d4491/synapse/util/metrics.py#L112-L119
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Store the JSON content of scheduled delayed events as text instead of a
byte array. This brings it in line with the `event_json` table's `json`
column, and fixes the inability to schedule a delayed event with
non-ASCII characters in its content.
Fixes#19242
Fixes https://github.com/element-hq/synapse/issues/19175
This PR moves tracking of what lazy loaded membership we've sent to each
room out of the required state table. This avoids that table from
continuously growing, which massively helps performance as we pull out
all matching rows for the connection when we receive a request.
The new table is only read when we have data in a room to send, so we
end up reading a lot fewer rows from the DB. Though we now read from
that table for every room we have events to return in, rather than once
at the start of the request.
For an explanation of how the new table works, see the
[comment](https://github.com/element-hq/synapse/blob/erikj/sss_better_membership_storage2/synapse/storage/schema/main/delta/93/02_sliding_sync_members.sql#L15-L38)
on the table schema.
The table is designed so that we can later prune old entries if we wish,
but that is not implemented in this PR.
Reviewable commit-by-commit.
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Rather than using dodgy regexes which keep breaking.
Also fixes a regression where it looks like we didn't fail CI if the
delta was in the wrong place.
This is useful as someone downstream can source the
`scripts-dev/complement.sh` script and run the same set of tests as
Synapse:
```bash
# Grab the test packages supported by Synapse.
#
# --fast: Skip rebuilding the docker images,
# --build-only: Will only build Docker images but because we also used `--fast`, it won't do anything.
# `>/dev/null` to redirect stdout to `/dev/null` to get rid of the `echo` logs from the script.
test_packages=$(source ${SYNAPSE_DIR}/scripts-dev/complement.sh --fast --build-only >/dev/null && echo "$SYNAPSE_SUPPORTED_COMPLEMENT_TEST_PACKAGES")
echo $test_packages
```
This is spawning from wanting to run the same set of Complement tests in
the https://github.com/element-hq/synapse-rust-apps project.
This is useful so that the script can be sourced by other scripts
without exiting the calling subshell (composable).
This is split out from https://github.com/element-hq/synapse/pull/19208
to make easy to understand PR's and build up to where we want to go.
The schema lint tries to make sure we don't add or remove indices in
schema files (rather than as background updates), *unless* the table was
created in the same schema file.
The regex to pull out the `CREATE TABLE` SQL incorrectly didn't
recognise `IF NOT EXISTS`.
There is a test delta file that shows that we accept different types of
`CREATE TABLE` and `CREATE INDEX` statements, as well as an index
creation that doesn't have a matching create table (to show that we do
still catch it). The test delta should be removed before merge.
Introduce `Clock.add_system_event_trigger(...)` to wrap system event
callback code in a logcontext, ensuring we can identify which server
generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.add_system_event_trigger(...)` over
`reactor.addSystemEventTrigger(...)`
Part of https://github.com/element-hq/synapse/issues/18905
Introduce `Clock.call_when_running(...)` to wrap startup code in a
logcontext, ensuring we can identify which server generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.call_when_running(...)` over
`reactor.callWhenRunning(...)`
Part of https://github.com/element-hq/synapse/issues/18905
