This PR adds a suite of tests for Synapse state events. It covers key
scenarios around creation, updates and state consistency to prevent
regressions in event processing and serialization.
The primary goal is to increase test coverage.
With `enable_set_displayname: false` (or `enable_set_avatar_url:
false`), refusing a profile change returned the right errcode with the
wrong status:
```
PUT /_matrix/client/v3/profile/@alice:example.com/displayname (displayname already set)
→ 400 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
DELETE /_matrix/client/v3/profile/@alice:example.com/displayname
→ 400 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
```
With this fix:
```
PUT /_matrix/client/v3/profile/@alice:example.com/displayname (displayname already set)
→ 403 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
DELETE /_matrix/client/v3/profile/@alice:example.com/displayname
→ 403 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
```
The spec defines the [403 response of `PUT
/_matrix/client/v3/profile/{userId}/{keyName}`](https://spec.matrix.org/v1.19/client-server-api/#put_matrixclientv3profileuseridkeyname)
as "The server is unwilling to perform the operation, either due to
insufficient permissions or **because profile modifications are
disabled**", while 400 is reserved for malformed input (`M_BAD_JSON`,
`M_MISSING_PARAM`, …).
Clients seem to rely on `errcode` field more than the HTTP Status Code,
that change seems safe.
Room topics are dropped from the search index whenever the
`event_search` background reindex runs (e.g. after a search index
rebuild, or when the background update is re-run on an upgraded
homeserver), making topics unsearchable even though the live write path
indexes them correctly.
The cause is a trailing comma in `_background_reindex_search`, which
turns the topic `value` into a 1-tuple instead of a string:
https://github.com/element-hq/synapse/blob/14c96c0f5444cbe28b6ac0361cf94216b8d352db/synapse/storage/databases/main/search.py#L211-L213
The downstream `if not isinstance(value, str): continue` guard then
silently skips *every* `m.room.topic` event, so no topic ever reaches
`event_search` during a reindex.
The regression was introduced in #18195, which added rich-text topic
support (MSC3765) to the reindex path.
### Problem Example
1. A room has topic "project roadmap".
2. An admin rebuilds the search index (or the `event_search` background
update re-runs).
3. `m.room.message` and `m.room.name` events are reindexed fine, but
every `m.room.topic` event is skipped.
4. Searching for "project roadmap" with key `content.topic` returns 0
results — the topic is permanently unsearchable until the event is sent
again.
---
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
For the Admin API /users endpoint, the response is meant to exclude the
approval flag when MSC3866 is enabled, but the filter that applies the
exclusion never actually got used. Fix things so that it does.
When a client (correctly) calls the `DELETE` endpoint to remove a custom
profile field (like Element X does with `m.status`), we incorrectly
don't include it in the sync response in legacy sync. This was due to
the fact that we cleaned up the sent fields down to what fields the
profile currently has.
Always ensure any fields in `ProfileUpdateAction.UPDATE` are sent down,
as `null` values for profile fields which have been deleted.
Fixes an issue where clearing a user status from Element X does not
reflect in the user status being cleared on Element Web.
Note, target is the v1.160.0 release branch due to customer commitments,
and this fixes web and mobile clients not working together correctly.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [ ] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Drive-by fix for a problem noticed when doing some other work.
This only triggers for 'custom'/generic fields, not the built-in
`displayname` and `avatar_url`.
Not seeing an open issue for it.
Return a 404, not 500, when looking up a profile field on a missing
profile
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This PR implements support for profile updates over Sliding Sync:
https://github.com/matrix-org/matrix-spec-proposals/pull/4262. This pr
may be easier to review as a whole than commit by commit.
This builds on the legacy sync profile updates feature
https://github.com/element-hq/synapse/pull/19556, specifically the
profile updates stream it added.
Submitting for early review to get consensus on implementation. There
are some things we would like to add still, from spec, mainly:
* > Homeservers should only consider a profile field update "accepted"
by a client
> once the client returns with a new /sync request with the next /sync
token,
> NOT just after sending down the profile update. The client may never
receive
> response due to network conditions, or a bug in the client
implementation.
* > When a room enters this subset in this connection for the first
time, all requested
> fields from profiles of users in that room MAY be sent down. This
gives the client
> a base set of information for which future field updates can be
applied on top of.
> The homeserver MAY omit some fields and profiles if it believes that
the client has
> already received them, likewise repeat profiles MAY be sent down based
on homeserver
> implementation.
* > Finally, if the list of fields expands to cover a new field ID,
those fields should
> be sent down for all users that are within the current room subset.
Future incremental
> updates will then include changes to this field.
* Additionally, we would need to implement a lazy loading cache similar
to the legacy sync. (not part of MSC as such)
Depending on review these could either be added to this pr, or to keep
this pr from not growing too much, be added in a follow-up pr, as they
are more enhancement to this base sliding sync profile updates
functionality than a part of the core functionality.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>
Co-authored-by: Olivier 'reivilibre <oliverw@element.io>
We have an internal usage of `/scheduled_tasks` that would like to fetch
multiple actions at once (janitor).
We also make it so that invalid `status` values now return a 400 rather
than a 500.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
*Spawning from
https://github.com/element-hq/synapse/pull/19057#discussion_r2427537811,*
Fix tests that use `homeserver_to_use=GenericWorkerServer` not being
able to be run standalone
Fix https://github.com/element-hq/synapse/issues/15671 (previously
https://github.com/matrix-org/synapse/issues/15671)
Before this change:
```shell
$ poetry run trial tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
tests.storage.test_rollback_worker
WorkerSchemaTests
test_rolling_back ... [ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "synapse/tests/unittest.py", line 129, in new
return code(orig, *args, **kwargs)
File "synapse/tests/unittest.py", line 223, in setUp
return orig()
File "synapse/tests/unittest.py", line 398, in setUp
self.hs = self.make_homeserver(self.reactor, self.clock)
File "synapse/tests/storage/test_rollback_worker.py", line 54, in make_homeserver
hs = self.setup_test_homeserver(homeserver_to_use=GenericWorkerServer)
File "synapse/tests/unittest.py", line 669, in setup_test_homeserver
hs = setup_test_homeserver(
File "synapse/tests/server.py", line 1260, in setup_test_homeserver
prepare_database(
File "synapse/synapse/storage/prepare_database.py", line 167, in prepare_database
raise UpgradeDatabaseException(EMPTY_DATABASE_ON_WORKER_ERROR)
synapse.storage.prepare_database.UpgradeDatabaseException: Uninitialised database: run the main synapse process to prepare the database schema before starting worker processes.
tests.storage.test_rollback_worker.WorkerSchemaTests.test_rolling_back
-------------------------------------------------------------------------------
Ran 1 tests in 0.034s
FAILED (errors=1)
```
### What was the problem before?
[`PREPPED_SQLITE_DB_CONN`](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/tests/server.py#L1247-L1262)
is a process global and shared between all tests. Whichever test first
calls `setup_test_homeserver(...)` builds the template database for the
whole trial run.
`prepare_database(...)` has a built-in check to refuse upgrading the
database ["to avoid multiple workers doing it at
once."](https://github.com/element-hq/synapse/blob/1a1af7b622f219ba0f2501709298caa230ad3912/synapse/storage/prepare_database.py#L164-L167)
and throw `UpgradeDatabaseException`.
This means that if we happen to first run a test that uses a worker
(`homeserver_to_use=GenericWorkerServer`), `prepare_database(...)` will
just throw its `UpgradeDatabaseException`. And since
`PREPPED_SQLITE_DB_CONN` is assigned before `prepare_database(...)`, it
will never try to prepare again and the rest of the tests will fail.
The registration of `QuarantinedMediaStream` in
`ReplicationCommandHandler._streams_to_replicate` was missed when the
stream was added, so an instance configured as the
quarantined_media_changes stream writer never sent RDATA/POSITION for it
unless it was the main process.
Also add the stream to the `instance_map` config validation.
Stream was introduced in
https://github.com/element-hq/synapse/pull/19558
Fixes https://github.com/element-hq/synapse/issues/20080
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fix `RemoteJoinHelper` signing events with mismatched room version
compared to the `room_version` arg. The default room version on
`develop` is `11` but the `RemoteJoinHelper` `room_version` arg defaults
to `10` (room version mismatch). This mismatch wasn't present where this
fix was developed
(https://github.com/element-hq/synapse-private/pull/136) as the default
room version was only recently bumped to `11` via
https://github.com/element-hq/synapse/pull/18680 (not even in a release
yet).
Fixes the CI being broken on `develop` :x::
```
[ERROR]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/federation/test_federation_join_upgraded_room.py", line 298, in test_no_transfer_when_tombstone_does_not_match
join_helper.join(local_user_id, local_user_tok)
File "/home/runner/work/synapse/synapse/tests/federation/_remote_join.py", line 350, in join
self._test_case.helper.join(remote_room_id, local_user_id, tok=local_user_tok)
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 195, in join
return self.change_membership(
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 333, in change_membership
assert channel.code == expect_code, (
builtins.AssertionError: Expected: 200, got: 400, PUT /_matrix/client/r0/rooms/!remote-room:other.example.com/state/m.room.member/@user1:test?access_token=syt_dXNlcjE_JSEarRndiAGqtydnrdLn_33qlLD -> resp: b'{"errcode":"M_UNKNOWN","error":"No create event in state"}'
tests.federation.test_federation_join_upgraded_room.FederationJoinUpgradedRoomTestCase.test_no_transfer_when_tombstone_does_not_match
```
These tests were originally introduced
https://github.com/element-hq/synapse-private/pull/136 (developed
private as this was part of the Synapse security release) and introduced
into the public codebase via
https://github.com/element-hq/synapse/commit/cbc6934821aab314506c5957223c60c74eeda091
Fixes: https://github.com/element-hq/synapse/security/advisories/GHSA-vh4c-pqh4-w3wq
Fixes: https://github.com/matrix-org/internal-config/issues/1703
The key thing to understand is that in `synapse/util/httpresourcetree.py`,
we create `UnrecognizedRequestResource` and then dangle children (with real resources) off them.
Since `UnrecognizedRequestResource` returns itself as a catch-all 'dynamic child',
this means any `UnrecognizedRequestResource`s with real children can have unlimited path components inserted between it and its child.
So `/_matrix/INSERTED/static/client/login/style.css` or `/_matrix/INSERTED/AS/MANY/AS/I/WANT/static/client/login/style.css` would unexpectedly resolve to the resource.
Client, Federation and Admin APIs wouldn't have been affected because you wouldn't get through the regex routing that they use.
-----
Reviewed-on: https://github.com/element-hq/synapse-private/pull/143
The cache key of `_get_server_keys_json` is a single argument which is
itself a `(server_name, key_id)` tuple, and `store_server_keys_response`
passed that nested tuple straight into the cache invalidation stream.
psycopg2 quietly serialises the inner tuple as a Postgres *record*, so
the `keys` column of `cache_invalidation_stream_by_instance` ended up
holding the record literal as a single string (e.g.
`{"(srv,ed25519:abc)"}`) — which never matches the real cache key on the
receiving side, i.e. the
invalidation has always been a silent no-op on workers. The native Rust
backend's stricter parameter conversion turns the same nested tuple into
a loud `TypeError: unsupported parameter type for postgres: tuple`.
Fix it the same way https://github.com/element-hq/synapse/pull/18899 did
for `_get_e2e_cross_signing_signatures_for_device`, which has the same
nested-tuple key shape: invalidate the local cache directly, JSON-encode
the key for the replication row, and decode it again in
`process_replication_rows`.
Found as part of the the effort to port the database pool to Rust.
This PR uses the new `M_USER_LIMIT_EXCEEDED` common error code instead
of the malformed `M_RESOURCE_LIMIT_EXCEEDED` error code (as reported by
#18749) (spec: MSC4335).
The error code is also changed from `400` to `403` as this matches what
is in the spec for the endpoints:
https://spec.matrix.org/v1.18/client-server-api/#post_matrixmediav3upload
and
https://spec.matrix.org/v1.18/client-server-api/#put_matrixmediav3uploadservernamemediaid
(albeit the latter says that `M_FORBIDDEN` would be returned)
By default a new built-in `media_upload_limit_exceeded.html` template
will be served for the `info_uri`. Administrators can specify an
external URI in config instead.
Compatibility is retained for any existing modules making use the
`MediaUploadLimit` (e.g. via the `get_media_upload_limits_for_user`
callback).
guillermo