I *think* the intention is that `__all__` can be used to access all the
types that are exposed via the module API. However, some of them are
missing.
This PR adds those that are directly referenced but were not exposed.
There are some others that are indirectly referenced that I've not
included (e.g. `synapse.spam_checker_api.RegistrationBehaviour`).
When a client (correctly) calls the `DELETE` endpoint to remove a custom
profile field (like Element X does with `m.status`), we incorrectly
don't include it in the sync response in legacy sync. This was due to
the fact that we cleaned up the sent fields down to what fields the
profile currently has.
Always ensure any fields in `ProfileUpdateAction.UPDATE` are sent down,
as `null` values for profile fields which have been deleted.
Fixes an issue where clearing a user status from Element X does not
reflect in the user status being cleared on Element Web.
Note, target is the v1.160.0 release branch due to customer commitments,
and this fixes web and mobile clients not working together correctly.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [ ] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Drive-by fix for a problem noticed when doing some other work.
This only triggers for 'custom'/generic fields, not the built-in
`displayname` and `avatar_url`.
Not seeing an open issue for it.
Return a 404, not 500, when looking up a profile field on a missing
profile
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This PR implements support for profile updates over Sliding Sync:
https://github.com/matrix-org/matrix-spec-proposals/pull/4262. This pr
may be easier to review as a whole than commit by commit.
This builds on the legacy sync profile updates feature
https://github.com/element-hq/synapse/pull/19556, specifically the
profile updates stream it added.
Submitting for early review to get consensus on implementation. There
are some things we would like to add still, from spec, mainly:
* > Homeservers should only consider a profile field update "accepted"
by a client
> once the client returns with a new /sync request with the next /sync
token,
> NOT just after sending down the profile update. The client may never
receive
> response due to network conditions, or a bug in the client
implementation.
* > When a room enters this subset in this connection for the first
time, all requested
> fields from profiles of users in that room MAY be sent down. This
gives the client
> a base set of information for which future field updates can be
applied on top of.
> The homeserver MAY omit some fields and profiles if it believes that
the client has
> already received them, likewise repeat profiles MAY be sent down based
on homeserver
> implementation.
* > Finally, if the list of fields expands to cover a new field ID,
those fields should
> be sent down for all users that are within the current room subset.
Future incremental
> updates will then include changes to this field.
* Additionally, we would need to implement a lazy loading cache similar
to the legacy sync. (not part of MSC as such)
Depending on review these could either be added to this pr, or to keep
this pr from not growing too much, be added in a follow-up pr, as they
are more enhancement to this base sliding sync profile updates
functionality than a part of the core functionality.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>
Co-authored-by: Olivier 'reivilibre <oliverw@element.io>
The conversion of `device_lists_changes_in_room` rows into
`device_lists_outbound_pokes` is DB-bound on
`mark_redundant_device_lists_pokes`. The `UPDATE` uses the `(room_id,
stream_id)` index, so each call scans the unconverted backlog, getting
slower the further behind the conversion is.
Add a partial index matching the query via a background update, and skip
the (safe-to-skip) `UPDATE` until the index has been built. Also add a
metric reporting how far behind the conversion is, using the existing
`inserted_ts` column.
Fixes https://github.com/element-hq/backend-internal/issues/286
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Eric Eastwood <erice@element.io>
Follows: #20027
Noticed in https://github.com/element-hq/synapse/pull/20003
The problem was that `--no-root` prevents reinstalling the root project
(Synapse).
However, since we just did a `git checkout`, we need to reinstall the
root project in case the Rust code changed, as the
`poetry install` command is what causes the Rust to be recompiled.
Doing otherwise causes the 'Rust module outdated' error when importing
`synapse`.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Issue spotted in: https://github.com/element-hq/synapse/pull/20098
Follows: #20027
We already use `faketime` for Postgres, but I forgot that the SQLite
schema delta would have the same problem
and somehow tuned it out of the preview diff on the original PR.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
We have an internal usage of `/scheduled_tasks` that would like to fetch
multiple actions at once (janitor).
We also make it so that invalid `status` values now return a 400 rather
than a 500.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
We haven't done a full schema for about 20 schema versions, so no
surprise!
As a result, I want to produce a full schema soon.
In this PR I dust off `make_full_schema.sh` (which seems to have broken
after some SQLite changes)
and add a CI workflow that runs it (producing a diff) when someone
changes the schema.
The CI workflow also adds a sticky comment showing the diff on the
schema, so you can better appreciate the final effect of a change.
---
**Dead changes:**
I wanted to make it possible to generate a versioned full schema without
the manual work,
but you can't run the background updates without essentially starting up
a homeserver,
at which point it might fail because you haven't run all the deltas yet.
There's no actual good way to do this, short of deleting the latest
deltas (+ tweaking code to not crash without them)
or rolling back in the git history.
Backed out those changes, but they're preserved on the PR if interesting.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
*Spawning from
https://github.com/element-hq/synapse/pull/19057#discussion_r2427537811,*
Fix tests that use `homeserver_to_use=GenericWorkerServer` not being
able to be run standalone
Fix https://github.com/element-hq/synapse/issues/15671 (previously
https://github.com/matrix-org/synapse/issues/15671)
Before this change:
```shell
$ poetry run trial tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
tests.storage.test_rollback_worker
WorkerSchemaTests
test_rolling_back ... [ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "synapse/tests/unittest.py", line 129, in new
return code(orig, *args, **kwargs)
File "synapse/tests/unittest.py", line 223, in setUp
return orig()
File "synapse/tests/unittest.py", line 398, in setUp
self.hs = self.make_homeserver(self.reactor, self.clock)
File "synapse/tests/storage/test_rollback_worker.py", line 54, in make_homeserver
hs = self.setup_test_homeserver(homeserver_to_use=GenericWorkerServer)
File "synapse/tests/unittest.py", line 669, in setup_test_homeserver
hs = setup_test_homeserver(
File "synapse/tests/server.py", line 1260, in setup_test_homeserver
prepare_database(
File "synapse/synapse/storage/prepare_database.py", line 167, in prepare_database
raise UpgradeDatabaseException(EMPTY_DATABASE_ON_WORKER_ERROR)
synapse.storage.prepare_database.UpgradeDatabaseException: Uninitialised database: run the main synapse process to prepare the database schema before starting worker processes.
tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
-------------------------------------------------------------------------------
Ran 1 tests in 0.034s
FAILED (errors=1)
```
### What was the problem before?
[`PREPPED_SQLITE_DB_CONN`](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/tests/server.py#L1247-L1262)
is a process global and shared between all tests. Whichever test first
calls `setup_test_homeserver(...)` builds the template database for the
whole trial run.
`prepare_database(...)` has a built-in check to refuse upgrading the
database ["to avoid multiple workers doing it at
once."](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/synapse/storage/prepare_database.py#L164-L167)
and throw `UpgradeDatabaseException`.
This means that if we happen to first run a test that uses a worker
(`homeserver_to_use=GenericWorkerServer`), `prepare_database(...)` will
just throw its `UpgradeDatabaseException`. And since
`PREPPED_SQLITE_DB_CONN` is assigned before `prepare_database(...)`, it
will never try to prepare again and the rest of the tests will fail.
The registration of `QuarantinedMediaStream` in
`ReplicationCommandHandler._streams_to_replicate` was missed when the
stream was added, so an instance configured as the
quarantined_media_changes stream writer never sent RDATA/POSITION for it
unless it was the main process.
Also add the stream to the `instance_map` config validation.
Stream was introduced in
https://github.com/element-hq/synapse/pull/19558
Fixes https://github.com/element-hq/synapse/issues/20080
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Set `idle_in_transaction_session_timeout` on new postgres connections
Idle transactions can block maintenance tasks server-side like vacuums,
which can lead to bloat and performance issues.
We should never hit this timeout in normal operation, as Synapse should
always be actively using the connection when in a transaction and so it
should only ever be briefly idle. If we do hit this timeout, it's likely
that no progress is being made and so aborting the session is safe.
In certain cases we have seen connections leak, particularly when using
a connection pooler like pgcat, and this timeout will help with that.
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Hugh Nimmo-Smith