When a worker's replication of a stream stalls, clients whose previous
sync was served by a worker ahead of it hand us a token we will never
catch up to, `wait_for_stream_token` times out after 10s and `/sync`
returns an empty response indefinitely. Nothing metric-side pointed at
which stream had stalled (see #20080, where it was `quarantined_media`).
Count the timeouts against each stream the token is still ahead of us on,
and name those streams (and their positions) in the logs instead of
dumping two whole tokens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
We haven't done a full schema for about 20 schema versions, so no
surprise!
As a result, I want to produce a full schema soon.
In this PR I dust off `make_full_schema.sh` (which seems to have broken
after some SQLite changes)
and add a CI workflow that runs it (producing a diff) when someone
changes the schema.
The CI workflow also adds a sticky comment showing the diff on the
schema, so you can better appreciate the final effect of a change.
---
**Dead changes:**
I wanted to make it possible to generate a versioned full schema without
the manual work,
but you can't run the background updates without essentially starting up
a homeserver,
at which point it might fail because you haven't run all the deltas yet.
There's no actual good way to do this, short of deleting the latest
deltas (+ tweaking code to not crash without them)
or rolling back in the git history.
Backed out those changes, but they're preserved on the PR if interesting.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
*Spawning from
https://github.com/element-hq/synapse/pull/19057#discussion_r2427537811,*
Fix tests that use `homeserver_to_use=GenericWorkerServer` not being
able to be run standalone
Fix https://github.com/element-hq/synapse/issues/15671 (previously
https://github.com/matrix-org/synapse/issues/15671)
Before this change:
```shell
$ poetry run trial tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
tests.storage.test_rollback_worker
WorkerSchemaTests
test_rolling_back ... [ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "synapse/tests/unittest.py", line 129, in new
return code(orig, *args, **kwargs)
File "synapse/tests/unittest.py", line 223, in setUp
return orig()
File "synapse/tests/unittest.py", line 398, in setUp
self.hs = self.make_homeserver(self.reactor, self.clock)
File "synapse/tests/storage/test_rollback_worker.py", line 54, in make_homeserver
hs = self.setup_test_homeserver(homeserver_to_use=GenericWorkerServer)
File "synapse/tests/unittest.py", line 669, in setup_test_homeserver
hs = setup_test_homeserver(
File "synapse/tests/server.py", line 1260, in setup_test_homeserver
prepare_database(
File "synapse/synapse/storage/prepare_database.py", line 167, in prepare_database
raise UpgradeDatabaseException(EMPTY_DATABASE_ON_WORKER_ERROR)
synapse.storage.prepare_database.UpgradeDatabaseException: Uninitialised database: run the main synapse process to prepare the database schema before starting worker processes.
tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
-------------------------------------------------------------------------------
Ran 1 tests in 0.034s
FAILED (errors=1)
```
### What was the problem before?
[`PREPPED_SQLITE_DB_CONN`](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/tests/server.py#L1247-L1262)
is a process global and shared between all tests. Whichever test first
calls `setup_test_homeserver(...)` builds the template database for the
whole trial run.
`prepare_database(...)` has a built-in check to refuse upgrading the
database ["to avoid multiple workers doing it at
once."](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/synapse/storage/prepare_database.py#L164-L167)
and throw `UpgradeDatabaseException`.
This means that if we happen to first run a test that uses a worker
(`homeserver_to_use=GenericWorkerServer`), `prepare_database(...)` will
just throw its `UpgradeDatabaseException`. And since
`PREPPED_SQLITE_DB_CONN` is assigned before `prepare_database(...)`, it
will never try to prepare again and the rest of the tests will fail.
The registration of `QuarantinedMediaStream` in
`ReplicationCommandHandler._streams_to_replicate` was missed when the
stream was added, so an instance configured as the
quarantined_media_changes stream writer never sent RDATA/POSITION for it
unless it was the main process.
Also add the stream to the `instance_map` config validation.
Stream was introduced in
https://github.com/element-hq/synapse/pull/19558
Fixes https://github.com/element-hq/synapse/issues/20080
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Set `idle_in_transaction_session_timeout` on new postgres connections
Idle transactions can block maintenance tasks server-side like vacuums,
which can lead to bloat and performance issues.
We should never hit this timeout in normal operation, as Synapse should
always be actively using the connection when in a transaction and so it
should only ever be briefly idle. If we do hit this timeout, it's likely
that no progress is being made and so aborting the session is safe.
In certain cases we have seen connections leak, particularly when using
a connection pooler like pgcat, and this timeout will help with that.
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Broken out of #20027 because I'd like to have it land first.
> It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
>
> We haven't done a full schema for about 20 schema versions, so no
surprise!
>
> As a result, I want to produce a full schema soon.
When running `make_full_schema.sh`, these drop statements now cause the
error:
```
Parse error near line 2: table event_search_content may not be dropped
Parse error near line 3: table event_search_segments may not be dropped
Parse error near line 4: table event_search_segdir may not be dropped
Parse error near line 5: table event_search_docsize may not be dropped
Parse error near line 6: table event_search_stat may not be dropped
Parse error near line 7: table user_directory_search_content may not be dropped
Parse error near line 8: table user_directory_search_segments may not be dropped
Parse error near line 9: table user_directory_search_segdir may not be dropped
Parse error near line 10: table user_directory_search_docsize may not be dropped
Parse error near line 11: table user_directory_search_stat may not be dropped
```
It seems SQLite has cracked down on code that edits the internal tables.
Because SQLite dumps the schema with `CREATE TABLE IF NOT EXISTS` for
these virtual tables, it's harmless to leave them in the schema dump.
---------
Signed-off-by: Olivier 'reivilibre' <oliverw@matrix.org>
To make it easy to share to people and aid in debugging when they run into
performance issues.
Unfortunately, the Grafana UI doesn't make the export or import steps
easy so it involves some manual Grafana API calls.
As first explored in
https://github.com/element-hq/synapse-rust-apps/pull/397
Without this change, it was linking to `develop` as we were on the
`develop` branch because of the previous `merge-back` step at this point
in the release process.
Follow-up to https://github.com/element-hq/synapse/pull/19984 as this
was an oversight and I assumed we would still be on the `release-v1.158`
branch by that point.
Fix `RemoteJoinHelper` signing events with mismatched room version
compared to the `room_version` arg. The default room version on
`develop` is `11` but the `RemoteJoinHelper` `room_version` arg defaults
to `10` (room version mismatch). This mismatch wasn't present where this
fix was developed
(https://github.com/element-hq/synapse-private/pull/136) as the default
room version was only recently bumped to `11` via
https://github.com/element-hq/synapse/pull/18680 (not even in a release
yet).
Fixes the CI being broken on `develop` :x::
```
[ERROR]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/federation/test_federation_join_upgraded_room.py", line 298, in test_no_transfer_when_tombstone_does_not_match
join_helper.join(local_user_id, local_user_tok)
File "/home/runner/work/synapse/synapse/tests/federation/_remote_join.py", line 350, in join
self._test_case.helper.join(remote_room_id, local_user_id, tok=local_user_tok)
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 195, in join
return self.change_membership(
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 333, in change_membership
assert channel.code == expect_code, (
builtins.AssertionError: Expected: 200, got: 400, PUT /_matrix/client/r0/rooms/!remote-room:other.example.com/state/m.room.member/@user1:test?access_token=syt_dXNlcjE_JSEarRndiAGqtydnrdLn_33qlLD -> resp: b'{"errcode":"M_UNKNOWN","error":"No create event in state"}'
tests.federation.test_federation_join_upgraded_room.FederationJoinUpgradedRoomTestCase.test_no_transfer_when_tombstone_does_not_match
```
These tests were originally introduced
https://github.com/element-hq/synapse-private/pull/136 (developed
private as this was part of the Synapse security release) and introduced
into the public codebase via
https://github.com/element-hq/synapse/commit/cbc6934821aab314506c5957223c60c74eeda091
This means we can point to a single source of truth instead of
duplicating the content to the tag and GitHub release. Less to manage
and worry about when you make some updates to the changelog (maintenance
burden). We also get to avoid the content sitting in the vendor lock-in
GitHub releases.
We point to
`https://github.com/element-hq/synapse/blob/{repo.active_branch.name}/CHANGES.md`
as it will have the relevant changelog entry at the top and won't change
as we archive releases on `develop`. Even for RC releases after the main
release goes out, the entry will still be towards the top. We could try
to get the heading anchor for the specific section but I thought that it
wasn't necessary (nice but more complex).
Erik Johnston