The new `sync/relations.rs` ports the relations handler's
`get_bundled_aggregations`: thread summaries with participation and
ignored users' replies subtracted, references, and the latest edit. The
Rust sync handler bundles them for limited and newly joined timelines,
loads the redactions of the bundled events, and the encoder serializes
them. Python no longer bundles the rooms Rust computes, and the room
entries hand Python the bundles only when something reads the timeline.
A thread whose latest reply is from an ignored user falls back to
Python for that room: finding the latest reply that is not takes the
relations pagination, which is not ported.
The relations queries read the database each time. Python's relations
caches are not mirrored.
Shadow mode now compares the Rust bundles with Python's, in the encoded
rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The rooms the Rust sync handler computes now reach the servlet as
`RustJoinedSyncResult` and `RustArchivedSyncResult`. They subclass the
Python entries, and their fields become Python objects only when read.
The servlet encodes them in one call to the new Rust `RoomsEncoder`,
which ports `encode_room` over the existing serializer. If that call
fails, Python encodes them.
The Rust handler loads each room's redactions as it computes the room.
The serializer reads redactions, bundles and module additions through a
`SerializeLookups` trait, so that it takes Rust events as well as Python
maps. Bundled aggregations are still computed in Python.
Shadow mode now also compares the encoded JSON of each room.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5b7Ruy4cgt27Mjq6Ptxbb
The first Rust database function to get a cache, and the first Python
`@cached` function to be replaced by one. The Rust `/versions` handler
calls `Store::is_feature_enabled_for_user` on every authenticated request,
and until now each call ran a query. The Python `is_feature_enabled` had a
cache of the same rows under another name, which every writer would have
had to invalidate alongside. Now there is one cache, keyed on
`(user_id, feature)`, and both sides read through it.
The main datastore builds the Rust `Store` in its `__init__`, over its own
`db_pool`, and `hs.get_rust_store()` hands it to the Rust handlers: the
datastore owns the store, the handlers borrow it. The store's
`is_feature_enabled_for_user` reaches Python as a callable cache object,
bound on the datastore under that name. Calling it answers a hit with an
already-fired Deferred, without leaving the reactor thread, and only a
miss goes through tokio. Being a `RustCache`, it is also what
`_invalidate_cache_and_stream`, replication and `register_cache` take, so
`set_features_for_user` invalidates it with the one call it already made,
and the name is spelled once in Rust, where the cache is built.
`is_feature_enabled` keeps its global-config short cut in Python and
delegates the per-user lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
These things are not needed now that MSC2965 has stabilised. In any
case, they
should never have been exposed without a config flag in the first place.
This speeds up the /createRoom calls a tiny bit by not checking the
senders push rules.
This helps speed up a lot of test runs in Element Web which
runs a lot of tests in one-person rooms (and creates a heck of a lot of rooms)
## Summary
- default `request_token_inhibit_3pid_errors` to `true` to prevent
account enumeration
- document the new default and the explicit opt-out
- add an upgrade note and regression coverage for the default
Fixes#18415.
Part of #19414 (Support Matrix 1.16).
⚠️ Before merging, wait for the following PRs to be merged first:
- https://github.com/element-hq/synapse/pull/20081
- https://github.com/element-hq/synapse/pull/20169
- https://github.com/element-hq/synapse/pull/20171
---
Since Matrix 1.16, the
[spec](https://spec.matrix.org/v1.19/client-server-api/#get_matrixclientv3sync)
defines the stable identifiers for
[MSC4222](https://github.com/matrix-org/matrix-spec-proposals/pull/4222):
the `use_state_after` query parameter on `GET /_matrix/client/v3/sync`,
and the `state_after` room response field:
> **`use_state_after`** — `boolean` — Controls whether to receive state
changes between the previous sync and the start of the timeline, or
between the previous sync and the end of the timeline. If this is set to
`true`, servers MUST respond with the state between the previous sync
and the end of the timeline in `state_after` and MUST omit `state`. […]
>
> **`state_after`** — […] Required if `use_state_after` was set to true
in the request, even if it is empty. Added in `v1.16`
>
> — [Matrix
Spec](https://spec.matrix.org/v1.19/client-server-api/#get_matrixclientv3sync)
Synapse so far only implemented the unstable identifiers
(`org.matrix.msc4222.use_state_after` /
`org.matrix.msc4222.state_after`), gated behind the `msc4222_enabled`
experimental flag and the per-user experimental features admin API.
This PR:
- accepts the stable `use_state_after` query parameter, unconditionally,
and responds with the stable `state_after` field. The unstable query
parameter is kept for a transition period; clients opting in with it get
the unstable response field name back, so unstable-era clients see an
unchanged wire format. If both parameters are set, the stable one wins.
- removes the `msc4222_enabled` experimental flag and the `msc4222`
per-user feature from the experimental features admin API, since the
behaviour is no longer gated.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
TLDR: `PUT` on a server-default push rule now applies the same existence
check that `GET` already does. A rule that is not served cannot be
modified.
Synapse ships some server-default push rules behind experimental feature
flags, so the set of server-default rules a user has depends on the
server's configuration. Most flags add rules that only exist while the
flag is on.
[MSC4210](https://github.com/matrix-org/matrix-spec-proposals/pull/4210)
works the other way: it removes the three legacy mention rules while its
flag is on. The flags that change the set are:
- `msc1767_enabled`:
[MSC1767](https://github.com/matrix-org/matrix-spec-proposals/pull/1767)
extensible events rules
- `msc3381_polls_enabled`:
[MSC3381](https://github.com/matrix-org/matrix-spec-proposals/pull/3381)
poll rules
- `msc3664_enabled`:
[MSC3664](https://github.com/matrix-org/matrix-spec-proposals/pull/3664)
reply rule
- `msc4028_push_encrypted_events`:
[MSC4028](https://github.com/matrix-org/matrix-spec-proposals/pull/4028)
encrypted event rule
- `msc4306_enabled`:
[MSC4306](https://github.com/matrix-org/matrix-spec-proposals/pull/4306)
thread subscription rules
- `msc4210_enabled`: removes `.m.rule.contains_display_name`,
`.m.rule.contains_user_name` and `.m.rule.roomnotif`
`GET /pushrules/` only lists the rules that exist for that
configuration. `GET` on a single rule that is not in the set returns
`404 M_NOT_FOUND`. The same set is used to evaluate notifications, and
to build the `m.push_rules` account data in `/sync`.
`PUT .../enabled` and `PUT .../actions` did not use that set. They
accepted any rule ID that Synapse knows about, whether or not the server
currently serves it. Clients could modify a rule the server had just
said did not exist. The request returned `200`, the change was stored,
and it was then ignored. Both endpoints now return `404 M_NOT_FOUND` for
such a rule, as [the spec
requires](https://spec.matrix.org/v1.19/client-server-api/#put_matrixclientv3pushrulesglobalkindruleidenabled)
for a rule that does not exist.
Before (`msc4210_enabled: true`):
```
GET /_matrix/client/v3/pushrules/global/override/.m.rule.roomnotif
404 {"errcode": "M_NOT_FOUND", "error": "..."}
PUT /_matrix/client/v3/pushrules/global/override/.m.rule.roomnotif/enabled
{"enabled": false}
200 {}
```
After:
```
GET /_matrix/client/v3/pushrules/global/override/.m.rule.roomnotif
404 {"errcode": "M_NOT_FOUND", "error": "..."}
PUT /_matrix/client/v3/pushrules/global/override/.m.rule.roomnotif/enabled
{"enabled": false}
404 {"errcode": "M_NOT_FOUND", "error": "Unknown rule"}
```
This is a no-op for default configurations. All of these flags are off
by default, and rules that are served are still modifiable as before.
Use a new, dedicated endpoint for scheduling delayed events. This should
deprecate the `delay` query parameter on the `/send` and `/state`
endpoints.
Co-authored-by: Paul Chobert <paul@chobert.fr>
Co-authored-by: Eric Eastwood <erice@element.io>
In a similar vein to #20315, this attempts to speed up Synapse when
delayed events aren't in use. Here we skip doing any work if the room
event stream has not advanced.
Fixes: #19661
Part of: MSC4354
First introduced some helpers for querying a sharded stream, which were moved to
https://github.com/element-hq/synapse/pull/20108.
- SQL builder for 'give me rows between two sharded tokens'
- a function to return the 'new' from_token after a limited read
Then make the sticky events portions of both oldschool and sliding sync use
sharded tokens.
This prevents a 'torn sync' problem where, because the events and sticky
events streams are both sharded with the same writers:
1. persisting a sticky event leads to an event row and a `sticky_events`
row being created for event `e` (on writer `W`).
2. Suppose another writer is still persisting, so the position of the
stream for writer W advances, but the global
['linear'](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html?highlight=%E2%80%9Clinear%E2%80%9D#current-stream-id)
(I have been calling this 'baseline') position does not advance yet.
3. sync is sharding-aware for the regular events stream, so `e` appears
as a real-time timeline event, thus coming down sync
4. since sticky events were not sharding-aware, `e` is not detected as a
sticky event in this sync response. The sticky event position doesn't
advance in the sync token, either.
5. (assume the position advances because other writers have finished
writing and so the 'linear' position advances for the sticky events
stream)
6. the client then syncs again, using the new sync token. This time, `e`
appears in the sticky event section.
If we make sticky events sharding-aware in sync, then `e` would appear
for consideration in both the timeline section **and** the sticky event
section in the first sync.
It then gets deduplicated out of the sticky section since it is
appearing in the timeline section in the same sync.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
In `/sync` there is a mode called lazy loaded (LL) members. In this mode
`/sync` does not return all memberships in the room, instead it only
returns a) membership changes, and b) the memberships of senders of
events in the timeline (if not previously sent).
Synapse implements a lazy-loaded member cache to avoid repeatedly
sending down the same memberships when a single user sends multiple
events. It does this by storing for each user/device the memberships it
has previously returned in an LRU cache. Note that it is fine for the
server to return the same membership, clients handle that fine, but it
is a waste of bandwidth.
The current lazy-loaded membership cache has three intertwined flaws:
1. It does not handle retries or clients re-using old tokens.
2. It incorrectly applies the membership cache to membership changes,
not just memberships added by lazy-loading.
3. It does not correctly handle multiple "sync streams" from the same
device.
In this PR we explicitly fix the first two issues in two commits, and
the third issue is mitigated:
1. Alongside each membership in the LL cache we store the `next_batch`
position of the response that carried it. A client that processed a
response syncs from its `next_batch` next. When a request arrives with a
`since` token, we delete every membership recorded after that token, as
they were sent in responses the client never processed. Memberships
recorded at or before `since` are kept.
2. We always return memberships that have changed, even if they appear
in the LL member cache.
These changes also half mitigate the issue of running multiple `/sync`
loops per-device (though we do not see this in the wild), as we'd at
least always return the changed membership states.
Fixes#19978 and #20278, supersedes #20279
Fixes#15871
### What was broken
Sending `{"auth": null}` to any endpoint that requires user-interactive
auth (for example `POST /keys/device_signing/upload`, `DELETE
/devices/{deviceId}` or `POST /register`) returned a 500 instead of the
usual 401 with the available flows. A non-object `auth` on endpoints
that do not validate the body with a model (such as `/register`) could
also 500, e.g. `{"auth": ["session"]}`.
### Root cause
`AuthHandler.check_ui_auth` did `clientdict.pop("auth", {})`, so the
`{}` default only applies when the key is missing. An explicit `null`
came through as `None` and the following `"session" in authdict` raised
`TypeError`. `get_session_id` had the same assumption that `auth` is a
dict.
### Pull Request Checklist
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Signed-off-by: Ankit Jha <ankit.jha@tradomate.one>
This pull request fixes a few issues with the profile update stream,
when a user leaves a room. This is basically just mimicking what we
already had for when someone leaves a room, and they no longer share any
rooms, but in reverse - ie when we leave a room, and no longer share
rooms with some users. This was missed in the implementation when adding
the profile update stream for legacy and sliding sync.
[Fix missing profile update stream rows on user leaving
room](https://github.com/element-hq/synapse/commit/e953322226731a99718ebe13139b9c5f05ae4b62)
When a user leaves a room, profile update stream rows are generated with
the `LEFT_ROOM` action for each user in the room that no longer shares a
room with the user who left the room.
This also needs to happen in reverse. The user who left the room needs
to have a profile update stream row for each user they no longer share a
room with.
If we don't do this, clients may keep stale data around even after they
don't share a room with a user, which may mean they don't know when to
refetch profiles after re-joining a room with the stale profile data
user.
[Clear out old profile update stream rows when leaving a
room](https://github.com/element-hq/synapse/commit/e7642bf141b098c1eefaf74655e5e7e18415500b)
When we leave a room, ensure profile update stream rows are cleared out
for every user we no longer share a room with. This is the same as what
happens when someone else leaves a room, but in reverse. The only
remaining profile update stream row should be the `LEFT_ROOM` action.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
This is to prevent the case where URL previews return a MXC that
immediately 404s because the media in question has been quarantined.
This is mostly to help implementations which currently show an ugly
empty preview due to the MXC being sent down, despite being invalid.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
[MSC4140](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/4140-delayed-events-futures.md),
"Account deactivation":
> when an account is deactivated, the homeserver MUST cancel that
account's delayed events which have not yet been added to a room's event
DAG. These cancelled records MAY be removed immediately, including their
stored event content, as an exception to the usual finalised-record
retention policy.
Deactivation currently leaves the user's scheduled delayed events,
including their content, in place, and they are still attempted at their
scheduled time:
| Delayed event scheduled by the user | What happens after deactivation
today |
| --- | --- |
| Message or state event, fires after deactivation has made the user
leave the room | The send fails with a 403 (user not in room) and the
record is dropped |
| Message or state event, fires before deactivation has made the user
leave the room (leaving happens room by room in the background) | The
event is sent |
| `m.room.member` join for themselves in a public room | The deactivated
user re-joins the room |
### What changes
- Deactivating an account (client or admin API) removes the user's
unsent delayed events as its first step, and re-arms the send timer for
whatever is scheduled next.
- Deactivation can run on a worker, while the send timer lives on the
main process, so the cancellation goes through a new replication
request.
- A delayed event whose send has already started is left alone, as with
a normal cancel: it is already on its way into the DAG, and the send
path removes its record itself.
Suspension and locking are unchanged.
### After #19038
Today a cancelled delayed event is simply deleted, so this PR deletes
the user's records too. #19038 changes cancellation to keep the record
and mark it as cancelled ("finalised"), so that clients can look up what
happened to a delayed event. Once it has landed, deactivation should
probably finalise the user's records as cancelled the same way, rather
than delete them. The MSC allows either: it permits removing the records
immediately, content included, as an exception to the usual retention of
finalised records, which is worth doing at least when the user asks to
be erased.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Ferrazzutti <af_0_af@hotmail.com>
Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
Previously the tokio runtime was stashed in a hidden attribute on the
reactor object, installed lazily by whichever Rust code first needed it,
and started via `callWhenRunning`.
Instead, we create a `RustRuntime` (accessible via
`HomeServer.get_rust_runtime()`) that holds any per-reactor Rust state,
such as the tokio runtime. It is constructed lazily on use. Rust
consumers (`HttpClient`, `VersionsHandler`, the Python DB pool wrapper)
now receive the runtime or reactor handle explicitly, and the
`reactor.run()` / manual-startup workarounds in tests are no longer
needed.
We also add helper wrappers in Rust for `Reactor` and `HomeServer` that
exposes the needed functionality.
The aim is to allow us to have a Rust-side clock (mainly to get the
current time), that respects the unit test per-reactor time management.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR fixes the issue described as comment here:
https://github.com/element-hq/synapse/issues/18793#issuecomment-3502202379
In Element Call, this shows up as ghost participants: someone who left
the call keeps being displayed until a later state change refreshes the
room.
The bug is not specific to Element Call: any state event can be
affected, RTC membership just changes often enough to make it visible.
## What happens
Alice has a client syncing against a homeserver where events are
persisted by one worker (the event persister) and `/sync` is served by
another (the sync worker). Her client is parked in a long-poll: `GET
/sync?since=s99&timeout=30000`.
Bob joins a call at the same moment Carol sends a message. Carol's
message reaches the persister first; Bob's `m.call.member` arrives while
that write is still in flight, so the per-room persist queue groups them
into one transaction:
```
events (each gets its own stream ordering):
stream_ordering 100: m.room.message Carol
stream_ordering 101: m.call.member Bob (state)
current_state_delta_stream (how state_after finds state changes):
stream_id 100 ────► (m.call.member, @bob) -> $bob_join_call
▲
└─ stamped with the batch MINIMUM (100), not the event's own 101
(see `_update_current_state_txn`)
```
The transaction commits: both events and the delta row are now in the
database, atomically.
The persister then announces the new events over replication, one RDATA
token per stream ordering — rows are only merged into one token when
they share a position, and 100 and 101 don't. So the sync worker's
events-stream position steps 99 → 100 → 101, and on reaching 100 it
pokes the notifier.
Alice's long-poll wakes at exactly that moment. Her response is built at
the worker's *current* position — `end = 100` — with RDATA 101 still in
the queue:
```
Sync A (since=99, end=100):
timeline: events 99 < ordering ≤ 100 → [Carol's message]
state_after: deltas 99 < stream_id ≤ 100 → [$bob_join_call] ← delivered EARLY
next_batch: s100 ← mid-batch token
```
No race on the client's side is needed: the server *hands out* the
mid-batch token as `next_batch`. Alice's client re-polls with it, as
every sync client does. The worker has meanwhile processed RDATA 101:
```
Sync B (since=100, end=101):
timeline: events 100 < ordering ≤ 101 → [Bob's m.call.member @101] ✓
state_after: deltas 100 < stream_id ≤ 101 → [] row is stamped 100 ✗
```
A state event in the timeline with an empty `state_after`. An MSC4222
client trusts `state_after` over timeline state events, so Alice's copy
of Bob's call membership never updates from this response.
On a single process this cannot happen: the batch's stream IDs are
released as a whole, so the position visible to `/sync` jumps 99 → 101
and `s100` is never handed out. Only a process that learns its position
from replication — any sync worker — ticks through the middle of a
batch.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Instead, treat them as absent fields as they feel like they should be.
The database implementation detail that these fields have a dedicated
column with `NULL`
when unset is kept to the storage layer.
The goal here is to reduce the amount of special casing needed for these
two original profile fields and treat them a little bit more like
regular profile fields.
Follows: #20003
Follows: #20147 (needed as a bugfix to continue sending them down
oldschool sync when they get deleted. Without #20147, this PR would
break that — which matches how custom profile fields were broken too.)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Part of: #19414
When
[MSC4133](https://github.com/matrix-org/matrix-spec-proposals/pull/4133)
(custom profile fields) was implemented, the returned value for an unset
display name changed from `200 {}` to `200 { displayname: null }`. This
happened first on the unstable `uk.tcpip.msc4133` path in #17488
(1.123.0), then on the stable path when #18635 (1.135.0) unified the
`displayname`, `avatar_url` and custom field servlets. Neither PR
discussed the change in review, so it looks like an unintended side
effect of the refactor rather than a deliberate decision.
The v1.16 spec mandated to change from returning `200 {}` to `404` but
change was not identified as breaking and was eventually not implemented
in other clients and server. This PR has a sister MSC that proposes to
return to the pre-1.16 error codes:
[MSC4537](https://github.com/matrix-org/matrix-spec-proposals/pull/4537).
Before:
```
GET /_matrix/client/v3/profile/@alice:test/displayname
200 {"displayname": null}
```
After:
```
GET /_matrix/client/v3/profile/@alice:test/displayname
200 {}
```
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
The validation of `client_secret` params accepts invalid values: any
value that includes a char in `[0-9a-zA-Z.=_-]` is accepted, for example
`"café"` is accepted.
Instead synapse should only accept if all chars are within
`[0-9a-zA-Z.=_-]`, not just one.
This is the current validator:
```python
ClientSecretStr = Annotated[
str,
StringConstraints(
pattern="[0-9a-zA-Z.=_-]",
min_length=1,
max_length=255,
strict=True,
),
]
```
Unfortunately, Pydantic only defines the `pattern` argument as:
> ### pattern
> A regex pattern that the string must match.
Which is extremely imprecise.
Here is a little script to verify the behavior:
```python
from typing import Annotated
from pydantic import BaseModel, StringConstraints, ValidationError
def check(pattern: str) -> None:
ClientSecretStr = Annotated[
str,
StringConstraints(pattern=pattern, min_length=1, max_length=255, strict=True),
]
class Body(BaseModel):
client_secret: ClientSecretStr
try:
Body.model_validate({"client_secret": "café"})
print(f"pattern = {pattern!r}: 'café' ACCEPTED <-- should have been rejected")
except ValidationError:
print(f"pattern = {pattern!r}: 'café' rejected")
check("[0-9a-zA-Z.=_-]") # before the fix (unanchored)
check("^[0-9a-zA-Z.=_-]+$") # after the fix
```
which would output
```
pattern = '[0-9a-zA-Z.=_-]': 'café' ACCEPTED <-- should have been rejected
pattern = '^[0-9a-zA-Z.=_-]+$': 'café' rejected
```
## History
This is a regression of a previously reported and fixed bug:
- matrix-org/synapse#6766 (2020) reported that Synapse did not enforce
the spec's `client_secret` regex at all — with real-world fallout:
FluffyChat had started sending secrets containing `:` because nothing
rejected them. Fixed by introducing `assert_valid_client_secret`
(matrix-org/synapse#6767).
- matrix-org/synapse#13188 (Synapse 1.66.0) ported the account endpoints
to Pydantic and transcribed the regex without anchors/quantifier;
Pydantic v1's `re.match` semantics meant only the *first* character was
validated.
- #19071 (Synapse 1.142.0) migrated to Pydantic v2, whose *search*
semantics weakened it further to "any one character anywhere".
---
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
Follow-up to #20149, which fixed the 500 when *getting* a profile field
for a user with no `profiles` row. The same crash was still reachable
when *setting* one via `PUT /_matrix/client/v3/profile/{userId}/{field}`
(as server admin). This PR splits that case in two:
* **The user exists but has no `profiles` row** (e.g. profile erased
upon deactivation):
* Before: `500 M_UNKNOWN` (`TypeError: cannot unpack non-sequence
NoneType` in the profile size check).
* After: `200`, the profile row is recreated with the field set.
* **The user does not exist at all**:
* Before: `500 M_UNKNOWN` (same crash).
* After: `404 M_NOT_FOUND`, without conjuring up an orphan profile row.
Fixing the crash also surfaced a latent SQLite-only bug where a field
set on a freshly created profile row was stored under the wrong key,
making it 404 on `GET` right after a successful `PUT`.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Part of: https://github.com/element-hq/synapse/issues/19415
Return `M_APPSERVICE_LOGIN_UNSUPPORTED` error code instead of the
unstable `IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTED` identifier.
> Servers MUST still allow application services to use the `/register`
endpoint with a login type of `m.login.application_service` even if they
don't support the Legacy Authentication API. In that case application
services MUST set the `"inhibit_login": true` parameter as they cannot
use it to log in as users. If the `inhibit_login` parameter is not set
to `true`, the server MUST return a 400 HTTP status code with an
`M_APPSERVICE_LOGIN_UNSUPPORTED` error code.
>
> [...]
>
> Application services MUST NOT use the `/login` endpoint if the server
doesn't support the Legacy authentication API. If `/login` is called
with the `m.login.application_service` login type the server MUST return
a 400 HTTP status code with an `M_APPSERVICE_LOGIN_UNSUPPORTED` error
code.
>
> — [Matrix v1.19, Application Service
API](https://spec.matrix.org/v1.19/application-service-api/#registration)
Synapse returns the correct 400 on both endpoints, but with the unstable
identifier.
Before:
```
POST /_matrix/client/v3/login {"type": "m.login.application_service", ...} # appservice with MSC4190 device management
POST /_matrix/client/v3/register {"type": "m.login.application_service", ...} # without "inhibit_login": true
400 {"errcode": "IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTED"}
```
After:
```
POST /_matrix/client/v3/login {"type": "m.login.application_service", ...} # appservice with MSC4190 device management
POST /_matrix/client/v3/register {"type": "m.login.application_service", ...} # without "inhibit_login": true
400 {"errcode": "M_APPSERVICE_LOGIN_UNSUPPORTED"}
```
A Sister PR exists in MAS:
https://github.com/element-hq/matrix-authentication-service/pull/5961 ;
when delegation is enabled, `/login` reaches MAS instead of Synapse, and
MAS currently answers `m.login.application_service` with `M_UNKNOWN`.
Part of https://github.com/element-hq/synapse/issues/18118
## What the spec says
Since Matrix v1.13 (introduced by
[MSC4178](https://github.com/matrix-org/matrix-spec-proposals/pull/4178)),
the `400` response of [`POST
/_matrix/client/v3/account/3pid/email/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3account3pidemailrequesttoken)
lists, among the "Error codes that can be returned":
> - `M_THREEPID_MEDIUM_NOT_SUPPORTED`: The homeserver does not support
adding email addresses.
> - `M_INVALID_PARAM`: The email address given was not valid.
and [`POST
/_matrix/client/v3/account/3pid/msisdn/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3account3pidmsisdnrequesttoken)
likewise:
> - `M_THREEPID_MEDIUM_NOT_SUPPORTED`: The homeserver does not support
adding phone numbers.
> - `M_INVALID_PARAM`: The phone number given was not valid.
## What was missing
Synapse implemented the headline case (unsupported medium), but around
it:
- A malformed email address or country code was reported with the
generic `M_BAD_JSON` instead of `M_INVALID_PARAM`. The email validator
deliberately kept `M_BAD_JSON` "to ensure backward compatibility of HTTP
error codes" (matrix-org/synapse#13687, 2022) — that predates Matrix
v1.13, which now lists `M_INVALID_PARAM` for this case.
- On the msisdn variant, the unsupported-medium check ran after the
denied/in-use checks, so a request wrong in two ways reported the other
fault; the email variant checks it first.
## What this PR changes
- Malformed email addresses and country codes on
`/account/3pid/{email,msisdn}/requestToken` are reported with
`M_INVALID_PARAM`. Both flow through the existing errcode translation as
`value_error`: the email validator raises a plain `ValueError`, and the
country-code constraint (`ISO3166_1_Alpha_2`) declares its own error via
pydantic-core's `custom_error_schema`.
- On the msisdn variant the unsupported-medium check now runs before the
denied/in-use checks, as on the email variant.
- The country-code type is renamed from `ISO3116_1_Alpha_2` to
`ISO3166_1_Alpha_2` (typo in the standard's number).
[`/account/password/email/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken)
(not covered by the v1.13 change) shares the email request body model,
so a malformed email there is now also reported with `M_INVALID_PARAM`
instead of `M_BAD_JSON`. Its `400` response is described as "the request
was invalid" and only names `M_SERVER_NOT_TRUSTED` explicitly ("can be
returned if…") rather than restricting the server to a fixed list, and
`M_INVALID_PARAM` is the spec's generic code for "A parameter that was
specified has the wrong value" ([other error
codes](https://spec.matrix.org/v1.19/client-server-api/#other-error-codes)).
Apply the `rc_reports` rate limit to the room reporting endpoint, [`POST
/_matrix/client/v3/rooms/{roomId}/report`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3roomsroomidreport)
(added in Matrix v1.13).
The spec marks this endpoint as **Rate-limited: Yes** (clients must
expect a `429 M_LIMIT_EXCEEDED`), and homeservers [SHOULD implement rate
limiting](https://spec.matrix.org/v1.19/client-server-api/#rate-limiting)
in general, but Synapse currently applies no limit here. The sibling
user reporting endpoint already uses `rc_reports`, so this reuses the
same limit instead of introducing a new config option.
Changes:
- Move the room report logic from `ReportRoomRestServlet` into a new
`ReportsHandler.report_room`, mirroring the existing `report_user`. The
rate limit is checked before the room existence lookup, so it bounds the
DB work a caller can trigger and cannot be used to tell existing rooms
from non-existing ones.
- The servlet keeps the existing behaviour of returning `200` regardless
of room existence when `msc4277_enabled` is set (the spec allows this
since v1.18).
- Add a regression test covering the `429` response and the per-user
rate limit override.
## The bug
With the experimental
[MSC4222](https://github.com/matrix-org/matrix-spec-proposals/pull/4222)
implementation enabled (`use_state_after`) and lazy-loading of room
members, an incremental `/sync` could disclose state from **after** the
user's leave in a left room's `state_after`.
1. Alice syncs with `lazy_load_members: true` and
`use_state_after=true`.
2. Bob sends a message in a room they share.
3. Alice leaves the room.
4. Bob updates his per-room displayname
5. Alice does an incremental sync covering steps 2–3. Alice's
`state_after` contains Bob's post-leave membership event from step 4
Alice should not see the new per-room display name of Bob.
## The fix
Copy what has been done for `_compute_state_delta_for_full_sync`: pass
`joined` down and, for rooms the user is no longer joined to, fetch the
memberships as of `end_token` via state groups (`get_state_ids_at`)
instead of current state.
With `enable_set_displayname: false` (or `enable_set_avatar_url:
false`), refusing a profile change returned the right errcode with the
wrong status:
```
PUT /_matrix/client/v3/profile/@alice:example.com/displayname (displayname already set)
→ 400 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
DELETE /_matrix/client/v3/profile/@alice:example.com/displayname
→ 400 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
```
With this fix:
```
PUT /_matrix/client/v3/profile/@alice:example.com/displayname (displayname already set)
→ 403 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
DELETE /_matrix/client/v3/profile/@alice:example.com/displayname
→ 403 {"errcode": "M_FORBIDDEN", "error": "Changing display name is disabled on this server"}
```
The spec defines the [403 response of `PUT
/_matrix/client/v3/profile/{userId}/{keyName}`](https://spec.matrix.org/v1.19/client-server-api/#put_matrixclientv3profileuseridkeyname)
as "The server is unwilling to perform the operation, either due to
insufficient permissions or **because profile modifications are
disabled**", while 400 is reserved for malformed input (`M_BAD_JSON`,
`M_MISSING_PARAM`, …).
Clients seem to rely on `errcode` field more than the HTTP Status Code,
that change seems safe.
For the Admin API /users endpoint, the response is meant to exclude the
approval flag when MSC3866 is enabled, but the filter that applies the
exclusion never actually got used. Fix things so that it does.
This PR implements support for profile updates over Sliding Sync:
https://github.com/matrix-org/matrix-spec-proposals/pull/4262. This pr
may be easier to review as a whole than commit by commit.
This builds on the legacy sync profile updates feature
https://github.com/element-hq/synapse/pull/19556, specifically the
profile updates stream it added.
Submitting for early review to get consensus on implementation. There
are some things we would like to add still, from spec, mainly:
* > Homeservers should only consider a profile field update "accepted"
by a client
> once the client returns with a new /sync request with the next /sync
token,
> NOT just after sending down the profile update. The client may never
receive
> response due to network conditions, or a bug in the client
implementation.
* > When a room enters this subset in this connection for the first
time, all requested
> fields from profiles of users in that room MAY be sent down. This
gives the client
> a base set of information for which future field updates can be
applied on top of.
> The homeserver MAY omit some fields and profiles if it believes that
the client has
> already received them, likewise repeat profiles MAY be sent down based
on homeserver
> implementation.
* > Finally, if the list of fields expands to cover a new field ID,
those fields should
> be sent down for all users that are within the current room subset.
Future incremental
> updates will then include changes to this field.
* Additionally, we would need to implement a lazy loading cache similar
to the legacy sync. (not part of MSC as such)
Depending on review these could either be added to this pr, or to keep
this pr from not growing too much, be added in a follow-up pr, as they
are more enhancement to this base sliding sync profile updates
functionality than a part of the core functionality.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>
Co-authored-by: Olivier 'reivilibre <oliverw@element.io>
