This partially fixes the bug
https://github.com/element-hq/synapse/issues/20116
Device list update EDUs from non-compliant (grandfathered historical)
user IDs are currently accepted over federation, stored, and surfaced to
clients in `/sync`'s `device_lists.changed` array.
> For current room versions, servers must still accept events using such
user IDs over federation; however they SHOULD NOT forward such user IDs
to clients when referenced outside the context of an event. For example,
device list updates from non-compliant user IDs would be dropped by the
receiving server.
>
> -- [Matrix
spec](https://spec.matrix.org/v1.14/appendices/#historical-user-ids),
clarified in Matrix v1.14 by
[matrix-spec#1506](https://github.com/matrix-org/matrix-spec/issues/1506)
### Problem Example
A remote server sends an `m.device_list_update` EDU for
`@héllo:remote.example` (non-ASCII localpart, outside the compliant
U+0021–U+007E range). Synapse:
- accepts and processes the update (resyncing the user's device list if
needed)
- stores it in the remote device list cache
- forwards `@héllo:remote.example` to local clients via
`device_lists.changed` in `/sync` (**the leak** — a non-compliant user
ID referenced outside event context)
---
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
This PR fixes the issue described as comment here:
https://github.com/element-hq/synapse/issues/18793#issuecomment-3502202379
In Element Call, this shows up as ghost participants: someone who left
the call keeps being displayed until a later state change refreshes the
room.
The bug is not specific to Element Call: any state event can be
affected, RTC membership just changes often enough to make it visible.
## What happens
Alice has a client syncing against a homeserver where events are
persisted by one worker (the event persister) and `/sync` is served by
another (the sync worker). Her client is parked in a long-poll: `GET
/sync?since=s99&timeout=30000`.
Bob joins a call at the same moment Carol sends a message. Carol's
message reaches the persister first; Bob's `m.call.member` arrives while
that write is still in flight, so the per-room persist queue groups them
into one transaction:
```
events (each gets its own stream ordering):
stream_ordering 100: m.room.message Carol
stream_ordering 101: m.call.member Bob (state)
current_state_delta_stream (how state_after finds state changes):
stream_id 100 ────► (m.call.member, @bob) -> $bob_join_call
▲
└─ stamped with the batch MINIMUM (100), not the event's own 101
(see `_update_current_state_txn`)
```
The transaction commits: both events and the delta row are now in the
database, atomically.
The persister then announces the new events over replication, one RDATA
token per stream ordering — rows are only merged into one token when
they share a position, and 100 and 101 don't. So the sync worker's
events-stream position steps 99 → 100 → 101, and on reaching 100 it
pokes the notifier.
Alice's long-poll wakes at exactly that moment. Her response is built at
the worker's *current* position — `end = 100` — with RDATA 101 still in
the queue:
```
Sync A (since=99, end=100):
timeline: events 99 < ordering ≤ 100 → [Carol's message]
state_after: deltas 99 < stream_id ≤ 100 → [$bob_join_call] ← delivered EARLY
next_batch: s100 ← mid-batch token
```
No race on the client's side is needed: the server *hands out* the
mid-batch token as `next_batch`. Alice's client re-polls with it, as
every sync client does. The worker has meanwhile processed RDATA 101:
```
Sync B (since=100, end=101):
timeline: events 100 < ordering ≤ 101 → [Bob's m.call.member @101] ✓
state_after: deltas 100 < stream_id ≤ 101 → [] row is stamped 100 ✗
```
A state event in the timeline with an empty `state_after`. An MSC4222
client trusts `state_after` over timeline state events, so Alice's copy
of Bob's call membership never updates from this response.
On a single process this cannot happen: the batch's stream IDs are
released as a whole, so the position visible to `/sync` jumps 99 → 101
and `s100` is never handed out. Only a process that learns its position
from replication — any sync worker — ticks through the middle of a
batch.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Fix#19905
Directly retrieve the `m.room.join_rules` event so the room summary data
is correct and matches `allowed_room_ids`. Otherwise there could be a
mis-match delay or the `join_rule` could be missing altogether.
Adds a `redis.username` config option.
Details:
A `username` without a `password` (or `password_path`) is refused at
startup. Redis has no wire form for a username without a password, and
txredisapi only sends `AUTH` when a password is set, so the username
would otherwise be silently ignored. An explicitly empty password is
accepted, since that is how a `nopass` ACL user is configured.
This relies on txredisapi 1.4.12, the first release to accept a
`username` kwarg. That upstream support was contributed by @karolyi
specifically to unblock this.
Fixes#19238.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Instead, treat them as absent fields as they feel like they should be.
The database implementation detail that these fields have a dedicated
column with `NULL`
when unset is kept to the storage layer.
The goal here is to reduce the amount of special casing needed for these
two original profile fields and treat them a little bit more like
regular profile fields.
Follows: #20003
Follows: #20147 (needed as a bugfix to continue sending them down
oldschool sync when they get deleted. Without #20147, this PR would
break that — which matches how custom profile fields were broken too.)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
A key refactoring for, and split out of,
https://github.com/element-hq/synapse/pull/20165
Would be easier to land first to isolate the diff.
Should be a standalone change with no behavioural change.
Motivation is that #20165 will round-robin between 'main queue'
transactions and 'sticky event' transactions.
To keep the data flow clear, I wanted to insert a typed struct (well,
`attrs` dataclass) as an interface between the 'preparation' of a
transaction and its 'completion'.
Doing this whilst keeping the asynchronous context manager style did not
lead to a readable result in my opinion.
(I would also say the async context manager is a touch 'magic' /
obscures control flow, but I suspect this is largely down to opinion.)
Replace _TransactionQueueManager with prepare/complete transaction
methods
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Part of: #19414
When
[MSC4133](https://github.com/matrix-org/matrix-spec-proposals/pull/4133)
(custom profile fields) was implemented, the returned value for an unset
display name changed from `200 {}` to `200 { displayname: null }`. This
happened first on the unstable `uk.tcpip.msc4133` path in #17488
(1.123.0), then on the stable path when #18635 (1.135.0) unified the
`displayname`, `avatar_url` and custom field servlets. Neither PR
discussed the change in review, so it looks like an unintended side
effect of the refactor rather than a deliberate decision.
The v1.16 spec mandated to change from returning `200 {}` to `404` but
change was not identified as breaking and was eventually not implemented
in other clients and server. This PR has a sister MSC that proposes to
return to the pre-1.16 error codes:
[MSC4537](https://github.com/matrix-org/matrix-spec-proposals/pull/4537).
Before:
```
GET /_matrix/client/v3/profile/@alice:test/displayname
200 {"displayname": null}
```
After:
```
GET /_matrix/client/v3/profile/@alice:test/displayname
200 {}
```
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Part of: #19415
TLDR: return the `M_UNKNOWN_DEVICE` error code instead of the unstable
`ORG.MATRIX.MSC4326.M_UNKNOWN_DEVICE` identifier.
> **[Added in `v1.17`]** Application services MAY similarly masquerade
as a specific device ID belonging the user ID through use of the
`device_id` query string parameter on the request. If the given device
ID is not known to belong to the user, the server will return a 400
`M_UNKNOWN_DEVICE` error.
>
> — [Matrix v1.19, Application Service API — Identity
assertion](https://spec.matrix.org/v1.19/application-service-api/#identity-assertion)
Synapse returns the correct 400, but with the unstable identifier.
MSC4326 was stabilized in Matrix 1.17 and its experimental flag was
already removed in #19033; only the error code identifier was left
behind.
Before:
```
GET /_matrix/client/v3/sync?user_id=@alice:test&device_id=NOT_A_REAL_DEVICE_ID # appservice token
400 {"errcode": "ORG.MATRIX.MSC4326.M_UNKNOWN_DEVICE"}
```
After:
```
GET /_matrix/client/v3/sync?user_id=@alice:test&device_id=NOT_A_REAL_DEVICE_ID # appservice token
400 {"errcode": "M_UNKNOWN_DEVICE"}
```
I verified that no implementation was currently handling the prefixed
error code.
Fixes: #20167
The `Schema Diff` workflow posts a PR comment showing the effective
schema diff. For PRs from forks, `GITHUB_TOKEN` is downgraded to
read-only, so the comment-posting step was silently failing.
### Changes
- `schema_diff.yml`: only post the comment directly when the PR is from
the same repository. For forked PRs, upload the diff (and PR number) as
a short-lived artifact instead of trying to comment.
- `schema_diff_comment.yml` (new): triggered by `workflow_run` after
`Schema Diff` completes, with `pull-requests: write` permission (granted
because this workflow always runs in the context of the base
repository). It downloads the artifact, if present, and posts the
comment on behalf of the forked PR.
This avoids `pull_request_target`, per the security concerns raised in
the issue (zizmor flags it as dangerous). The new workflow only ever
treats the downloaded artifact as inert comment text -- it is never
executed.
---------
Co-authored-by: Olivier 'reivilibre <oliverw@element.io>
Adds the serving functions needed for MSC4242: State DAGs. This PR adds
MSC4242 support to /make_join, /send_join and /get_missing_events, as
well as calculates the destinations for /send events correctly using
`prev_state_events`.
Built on top of https://github.com/element-hq/synapse/pull/19718 for the
storage functions it makes.
Split out from https://github.com/element-hq/synapse/pull/19425
Part of a series of 5x PRs to land the federation part of
[MSC4242](https://github.com/matrix-org/matrix-spec-proposals/pull/4242)
([storage](https://github.com/element-hq/synapse/pull/19718),
[fedclient](https://github.com/element-hq/synapse/pull/20127), serving
(this PR), inbound-joins, inbound-pulls).
Whilst this is mostly a port of the code in #19425 there are a few
changes:
- `/get_missing_events` accepts message events when walking the state
DAG, in which case it resolves the first hop to be that event's
`prev_state_events`. The original PR made the client `/event` the
message event and then set `latest=[prev_state_events]` on its own. This
is not very efficient (extra round trip to fetch the event) and there's
no reason why the server can't do the message->prev_state_events lookup,
so we do so. This matches the MSC examples.
- We cap the amount of events fetched via `/get_missing_events`. The MSC
allows it, so it's a good safety check.
- We sort the returned state DAG in `/send_join` by depth then event ID
so it's "mostly" sorted. This is more a formality than anything else,
the MSC does not mandate this, but it makes `/send_join` responses
deterministic.
- `notify_on_event_delivered_over_federation` is a new thing since
#19425, so we include state DAG events in it like we do with
state/auth_chain.
This PR does remove the forced `m.federate: false` setting for MSC4242
rooms, so it makes it possible for federated MSC4242 rooms to be made.
This is mostly so we can test via the endpoints. Given you must opt-in
to MSC4242 via the experimental features config option, it seems
reasonable to loosen this setting. The forced no-federation flag existed
prior to review saying that the MSC4242 room version could itself be
gated behind an experimental feature.
Reviewable commit-by-commit.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Eric Eastwood <erice@element.io>
This also changes the rendering of our custom helper `assertIncludes`.
Spawns from
https://github.com/element-hq/synapse/pull/20019#discussion_r3676539092
This PR hijacks `assertEqual` in order to substitute in our own error
rendering logic for set inequality.
(The motivation to do this is that we should just render sets in our
preferred style by default, without having to think about
`assertIncludes` with the `exact` flag or risk forgetting it.)
The error rendering for `assertIncludes` is adapted to make it reusable
in our `assertEqual` and to make it clearer _to me_ (I found it a bit
jarring that `+` was used more like a tick, when in other test
frameworks
I expect to see that as a diff marker).
I have tried to make it as clear as I could without being cryptic.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This solves the root cause of a user opening the thread panel in Element
Web DoSing synapse with recursive relation requests, starving out
delayed events and causing MatrixRTC calls to drop - see
https://github.com/matrix-org/matrix-js-sdk/pull/5519 for papering over
it clientside.
Fixes https://github.com/element-hq/synapse/issues/18788
Claude rationale:
Postgres cannot estimate the size of a recursive CTE. When it guesses
large
it stops probing events by event_id and instead hashes every event in
the
room, so a single recursive `/relations` request in a busy room takes
seconds
and a Threads-panel fan-out of 30 of them can pin a client-reader's DB
pool
for a minute. Joining events per recursion step keeps the lookups as
index
probes regardless of the estimate.
The recursion also moves from `UNION` to `UNION ALL`. An event carries a
single
`m.relates_to` and is stored as exactly one `event_relations` row
(unique index
on `event_id`), so the relation graph is a tree: every node is reached
along
one path and `UNION` never had duplicates to remove. `UNION ALL` drops
the
sort-and-dedupe pass over the working table on every iteration, which
matters more now that each row also carries the joined events columns. A
cycle from bogus events is still terminated by the depth bound, as
before,
and `UNION` gave no protection there anyway since such rows differ in
depth.
Measured on Postgres 16 against a synthetic corpus modelled on a large
homeserver: 4 rooms x 300k events; one 40-reply thread rooted 280k
events
back in the timeline, with 2 reactions per reply; every 5th event
elsewhere
a reaction, plus 200 popular roots with 2000 reactions each so that the
`relates_to_id statistics` are skewed the way they are in production.
Default
limit (6 rows), warm cache, JIT off:
```
before after
single request 77 ms 1.4 ms
20 concurrent requests 1.21 s 0.14 s
```
Before: Hash Join with a Hash over all 300k events of the room (4
batches).
After: Nested Loop with an Index Scan on `events_event_id_key` per row.
(found/solved by fable)
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Part of: https://github.com/element-hq/synapse/issues/19415
Follow-up to #19033
TLDR: date to drop support for prefixed `org.matrix.msc3202.device_id`
parameter was 2026-01-01. This PR removes the prefixed param.
Synapse has accepted the stable `device_id` parameter since v1.141.0
(#19033).
Before:
```
GET /_matrix/client/v3/account/whoami?user_id=@alice:test&org.matrix.msc3202.device_id=DEVICEID # appservice token
200 {"user_id": "@alice:test", "is_guest": false, "device_id": "DEVICEID"}
```
After:
```
GET /_matrix/client/v3/account/whoami?user_id=@alice:test&org.matrix.msc3202.device_id=DEVICEID # appservice token
200 {"user_id": "@alice:test", "is_guest": false}
```
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
The validation of `client_secret` params accepts invalid values: any
value that includes a char in `[0-9a-zA-Z.=_-]` is accepted, for example
`"café"` is accepted.
Instead synapse should only accept if all chars are within
`[0-9a-zA-Z.=_-]`, not just one.
This is the current validator:
```python
ClientSecretStr = Annotated[
str,
StringConstraints(
pattern="[0-9a-zA-Z.=_-]",
min_length=1,
max_length=255,
strict=True,
),
]
```
Unfortunately, Pydantic only defines the `pattern` argument as:
> ### pattern
> A regex pattern that the string must match.
Which is extremely imprecise.
Here is a little script to verify the behavior:
```python
from typing import Annotated
from pydantic import BaseModel, StringConstraints, ValidationError
def check(pattern: str) -> None:
ClientSecretStr = Annotated[
str,
StringConstraints(pattern=pattern, min_length=1, max_length=255, strict=True),
]
class Body(BaseModel):
client_secret: ClientSecretStr
try:
Body.model_validate({"client_secret": "café"})
print(f"pattern = {pattern!r}: 'café' ACCEPTED <-- should have been rejected")
except ValidationError:
print(f"pattern = {pattern!r}: 'café' rejected")
check("[0-9a-zA-Z.=_-]") # before the fix (unanchored)
check("^[0-9a-zA-Z.=_-]+$") # after the fix
```
which would output
```
pattern = '[0-9a-zA-Z.=_-]': 'café' ACCEPTED <-- should have been rejected
pattern = '^[0-9a-zA-Z.=_-]+$': 'café' rejected
```
## History
This is a regression of a previously reported and fixed bug:
- matrix-org/synapse#6766 (2020) reported that Synapse did not enforce
the spec's `client_secret` regex at all — with real-world fallout:
FluffyChat had started sending secrets containing `:` because nothing
rejected them. Fixed by introducing `assert_valid_client_secret`
(matrix-org/synapse#6767).
- matrix-org/synapse#13188 (Synapse 1.66.0) ported the account endpoints
to Pydantic and transcribed the regex without anchors/quantifier;
Pydantic v1's `re.match` semantics meant only the *first* character was
validated.
- #19071 (Synapse 1.142.0) migrated to Pydantic v2, whose *search*
semantics weakened it further to "any one character anywhere".
---
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
Application services that opt in to receiving ephemeral events
(`receive_ephemeral: true` in the
[registration](https://spec.matrix.org/v1.19/application-service-api/#registration),
added in Matrix v1.13 from
[MSC2409](https://github.com/matrix-org/matrix-spec-proposals/pull/2409))
are sent the private read receipts (`m.read.private`) of **every** user
in the rooms they are interested in — not just their own users.
The [Application Service
API](https://spec.matrix.org/v1.19/application-service-api/#pushing-ephemeral-data)
is explicit about this (*Pushing ephemeral data*, `m.receipt`):
> Private read receipts MUST only be sent for users matching one of the
application service's namespaces. Normal read receipts and threaded read
receipts are always sent.
This matches the [Client-Server
API](https://spec.matrix.org/v1.19/client-server-api/#private-read-receipts):
"Servers MUST NOT send the `m.read.private` receipt to any other user
than the one which originally sent it."
The restriction to namespaced users is safe because the appservice could
learn those receipts anyway by syncing as the user. For everyone else,
`m.read.private` exists precisely so that nobody — including bridges and
bots in the room — can observe it.
For example, take an appservice registered with:
```yaml
namespaces:
users:
- regex: "@_bridge_.*:example\\.org"
exclusive: true
```
When `@alice:example.org` (a regular user, not one of the appservice's)
and `@_bridge_bob:example.org` (a namespaced user) each send read
receipts in a bridged room, the appservice receives:
```json
{
"type": "m.receipt",
"room_id": "!room:example.org",
"content": {
"$event": {
"m.read": { "@alice:example.org": { "ts": 1436451550453 } },
"m.read.private": {
"@_bridge_bob:example.org": { "ts": 1436451550453 },
"@alice:example.org": { "ts": 1436451550453 }
}
}
}
}
```
- `m.read` from `@alice` — correct, public read receipts are always
sent.
- `m.read.private` from `@_bridge_bob` — correct, the user is within the
appservice's namespaces.
- `m.read.private` from `@alice` — **the leak**: their private read
receipt must not be sent to the appservice.
## History
- matrix-org/synapse#8437 (Oct 2020, Synapse 1.22.0) implemented MSC2409
ephemeral event delivery to appservices. No leak at that point: private
read receipts did not exist yet.
- matrix-org/synapse#10413 (Jul 2021, Synapse 1.40.0) added the initial,
experimental-flag-gated implementation of MSC2285 ("hidden" read
receipts) and filtered them out of the `/sync` path
(`filter_out_hidden`) — but not out of the appservice path in the same
file. This is where the leak originates, for servers with
`msc2285_enabled`.
- matrix-org/synapse#12168 (May 2022, Synapse 1.59.0) reworked this into
the `m.read.private` receipt type and `filter_out_private_receipts`; the
appservice path was again left unfiltered.
- matrix-org/synapse#13273 (Aug 2022, Synapse 1.65.0) moved to the
stable `m.read.private` identifier; the appservice path has leaked it
ever since.
---
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Follow-up to #20149, which fixed the 500 when *getting* a profile field
for a user with no `profiles` row. The same crash was still reachable
when *setting* one via `PUT /_matrix/client/v3/profile/{userId}/{field}`
(as server admin). This PR splits that case in two:
* **The user exists but has no `profiles` row** (e.g. profile erased
upon deactivation):
* Before: `500 M_UNKNOWN` (`TypeError: cannot unpack non-sequence
NoneType` in the profile size check).
* After: `200`, the profile row is recreated with the field set.
* **The user does not exist at all**:
* Before: `500 M_UNKNOWN` (same crash).
* After: `404 M_NOT_FOUND`, without conjuring up an orphan profile row.
Fixing the crash also surfaced a latent SQLite-only bug where a field
set on a freshly created profile row was stored under the wrong key,
making it 404 on `GET` right after a successful `PUT`.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Part of: https://github.com/element-hq/synapse/issues/19415
Return `M_APPSERVICE_LOGIN_UNSUPPORTED` error code instead of the
unstable `IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTED` identifier.
> Servers MUST still allow application services to use the `/register`
endpoint with a login type of `m.login.application_service` even if they
don't support the Legacy Authentication API. In that case application
services MUST set the `"inhibit_login": true` parameter as they cannot
use it to log in as users. If the `inhibit_login` parameter is not set
to `true`, the server MUST return a 400 HTTP status code with an
`M_APPSERVICE_LOGIN_UNSUPPORTED` error code.
>
> [...]
>
> Application services MUST NOT use the `/login` endpoint if the server
doesn't support the Legacy authentication API. If `/login` is called
with the `m.login.application_service` login type the server MUST return
a 400 HTTP status code with an `M_APPSERVICE_LOGIN_UNSUPPORTED` error
code.
>
> — [Matrix v1.19, Application Service
API](https://spec.matrix.org/v1.19/application-service-api/#registration)
Synapse returns the correct 400 on both endpoints, but with the unstable
identifier.
Before:
```
POST /_matrix/client/v3/login {"type": "m.login.application_service", ...} # appservice with MSC4190 device management
POST /_matrix/client/v3/register {"type": "m.login.application_service", ...} # without "inhibit_login": true
400 {"errcode": "IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTED"}
```
After:
```
POST /_matrix/client/v3/login {"type": "m.login.application_service", ...} # appservice with MSC4190 device management
POST /_matrix/client/v3/register {"type": "m.login.application_service", ...} # without "inhibit_login": true
400 {"errcode": "M_APPSERVICE_LOGIN_UNSUPPORTED"}
```
A Sister PR exists in MAS:
https://github.com/element-hq/matrix-authentication-service/pull/5961 ;
when delegation is enabled, `/login` reaches MAS instead of Synapse, and
MAS currently answers `m.login.application_service` with `M_UNKNOWN`.
Part of https://github.com/element-hq/synapse/issues/18118
## What the spec says
Since Matrix v1.13 (introduced by
[MSC4178](https://github.com/matrix-org/matrix-spec-proposals/pull/4178)),
the `400` response of [`POST
/_matrix/client/v3/account/3pid/email/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3account3pidemailrequesttoken)
lists, among the "Error codes that can be returned":
> - `M_THREEPID_MEDIUM_NOT_SUPPORTED`: The homeserver does not support
adding email addresses.
> - `M_INVALID_PARAM`: The email address given was not valid.
and [`POST
/_matrix/client/v3/account/3pid/msisdn/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3account3pidmsisdnrequesttoken)
likewise:
> - `M_THREEPID_MEDIUM_NOT_SUPPORTED`: The homeserver does not support
adding phone numbers.
> - `M_INVALID_PARAM`: The phone number given was not valid.
## What was missing
Synapse implemented the headline case (unsupported medium), but around
it:
- A malformed email address or country code was reported with the
generic `M_BAD_JSON` instead of `M_INVALID_PARAM`. The email validator
deliberately kept `M_BAD_JSON` "to ensure backward compatibility of HTTP
error codes" (matrix-org/synapse#13687, 2022) — that predates Matrix
v1.13, which now lists `M_INVALID_PARAM` for this case.
- On the msisdn variant, the unsupported-medium check ran after the
denied/in-use checks, so a request wrong in two ways reported the other
fault; the email variant checks it first.
## What this PR changes
- Malformed email addresses and country codes on
`/account/3pid/{email,msisdn}/requestToken` are reported with
`M_INVALID_PARAM`. Both flow through the existing errcode translation as
`value_error`: the email validator raises a plain `ValueError`, and the
country-code constraint (`ISO3166_1_Alpha_2`) declares its own error via
pydantic-core's `custom_error_schema`.
- On the msisdn variant the unsupported-medium check now runs before the
denied/in-use checks, as on the email variant.
- The country-code type is renamed from `ISO3116_1_Alpha_2` to
`ISO3166_1_Alpha_2` (typo in the standard's number).
[`/account/password/email/requestToken`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken)
(not covered by the v1.13 change) shares the email request body model,
so a malformed email there is now also reported with `M_INVALID_PARAM`
instead of `M_BAD_JSON`. Its `400` response is described as "the request
was invalid" and only names `M_SERVER_NOT_TRUSTED` explicitly ("can be
returned if…") rather than restricting the server to a fixed list, and
`M_INVALID_PARAM` is the spec's generic code for "A parameter that was
specified has the wrong value" ([other error
codes](https://spec.matrix.org/v1.19/client-server-api/#other-error-codes)).
Apply the `rc_reports` rate limit to the room reporting endpoint, [`POST
/_matrix/client/v3/rooms/{roomId}/report`](https://spec.matrix.org/v1.19/client-server-api/#post_matrixclientv3roomsroomidreport)
(added in Matrix v1.13).
The spec marks this endpoint as **Rate-limited: Yes** (clients must
expect a `429 M_LIMIT_EXCEEDED`), and homeservers [SHOULD implement rate
limiting](https://spec.matrix.org/v1.19/client-server-api/#rate-limiting)
in general, but Synapse currently applies no limit here. The sibling
user reporting endpoint already uses `rc_reports`, so this reuses the
same limit instead of introducing a new config option.
Changes:
- Move the room report logic from `ReportRoomRestServlet` into a new
`ReportsHandler.report_room`, mirroring the existing `report_user`. The
rate limit is checked before the room existence lookup, so it bounds the
DB work a caller can trigger and cannot be used to tell existing rooms
from non-existing ones.
- The servlet keeps the existing behaviour of returning `200` regardless
of room existence when `msc4277_enabled` is set (the spec allows this
since v1.18).
- Add a regression test covering the `429` response and the per-user
rate limit override.
## The bug
With the experimental
[MSC4222](https://github.com/matrix-org/matrix-spec-proposals/pull/4222)
implementation enabled (`use_state_after`) and lazy-loading of room
members, an incremental `/sync` could disclose state from **after** the
user's leave in a left room's `state_after`.
1. Alice syncs with `lazy_load_members: true` and
`use_state_after=true`.
2. Bob sends a message in a room they share.
3. Alice leaves the room.
4. Bob updates his per-room displayname
5. Alice does an incremental sync covering steps 2–3. Alice's
`state_after` contains Bob's post-leave membership event from step 4
Alice should not see the new per-room display name of Bob.
## The fix
Copy what has been done for `_compute_state_delta_for_full_sync`: pass
`joined` down and, for rooms the user is no longer joined to, fetch the
memberships as of `end_token` via state groups (`get_state_ids_at`)
instead of current state.
guillermo