Rust: `RoomVersion.msc4430_member_keys` and `MSC4430V12`
("org.matrix.msc4430.12" = `MSC4242V12` + the flag); redaction of
`m.room.member` keeps `content.domain` and drops `content.localpart` in such
rooms, so the reference hash, the event ID and the member key signature cover
the claimed domain but not the claimed localpart;
`membership_field::{DOMAIN,LOCALPART}`; `org.matrix.msc4430` in `/versions`;
`.pyi` updated. The version is defined but *not* registered in
`KNOWN_ROOM_VERSIONS`, not even when the flag is on.
Python: `MemberKey` in `synapse.types` next to the other identifier types, with
the grammar `^@[A-Za-z0-9_-]{43}$` (anything with a `:` is a user ID),
`is_member_key` (syntactic, regex only, for hot paths) and
`MemberKey.from_string` (strict: rejects non-canonical base64, so each public
key has exactly one member key string). `MemberKeys` constants
(`SIGNATURE_ENTITY = "sender"`, key ID `ed25519:1`) and
`EventContentFields.MSC4430_{LOCALPART,DOMAIN}`. `supports_msc4430_member_keys`
is a `TypeIs[MSC4430Event]` modelled on `supports_msc4242_state_dag`;
`MSC4430Event` subclasses `MSC4242Event`, since every member key room is a state
DAG room, so one predicate narrows to both.
Signing: `add_hashes_and_signatures_from_signers(room_version, event_dict,
signers, *, server_name)` hashes once and then applies `(entity, key)` signers
in order; `compute_member_key_room_signature(room_version, event_dict, signer,
*, server_name)` adds one signature to an already hashed event (the authoriser's
signature on `/send_join`, our co-signature on an inbound `/invite`) without
rehashing a PDU we have not checked. The three signer kinds are: the sender's
member key under `sender`; our own server name co-signing linkage information;
the authorising member key of a restricted join under the member key string
itself. Together these are the one choke point for signing in member key rooms,
and they refuse with `MemberKeySigningRefusedError` (500) any signer the MSC
does not allow: a `sender` signature by a key other than the one `sender` names,
a key ID other than `ed25519:1`, a server co-signature under any name but ours
or on anything but an `m.room.member` whose `content.domain` is us, or an
authorising signature on anything but a join that names that key in
`join_authorised_via_users_server`. A bug in choosing signers therefore cannot
make us sign as a key we do not hold or vouch for linkage information that is
not ours. An empty signer list hashes only (`/make_join` templates).
Loudness: `get_domain_from_id` and `get_localpart_from_id` raise
`MemberKeyHasNoDomainError` (a `SynapseError`, 500: it is a server bug) when
given a member key, on the no-colon branch only, so user IDs and aliases are
untouched. Because the HTTP layer logs every `SynapseError` at INFO without a
traceback, and the error may be caught before it gets there, both MSC4430 errors
log themselves at ERROR with the stack of the raise site: the point of them is
to name the unadapted code path. `hs.is_mine_id` is left alone: it is called on
raw client strings (state keys, sliding-sync `required_state`,
`/users/{id}/report`, module API) and on remote-controlled room content
(`/upgrade` moving a canonical alias), where "False if malformed" is relied upon
and a member-key-shaped string in a legacy room is just malformed.
`hs.is_mine_identity(room_id, identity)` is the sanctioned question for room
identities; it answers for user IDs and raises `MemberKeyHasNoDomainError` for
member keys. The Prometheus counter
`synapse_msc4430_unsupported_path_total{server_name,path}` and
`record_unsupported_path(server_name, path)` (counts always, warns once per path
per process) are for branches that deliberately skip member keys.
The identity module is `synapse/handlers/member_keys.py` (`MemberKeyHandler`,
`hs.get_member_key_handler()`). The `MemberKey` type and `is_member_key` live in
`synapse.types` because `get_domain_from_id` needs them and the module cannot
import handlers.
Config: `msc4430_enabled` implies `msc4428_enabled`; the Complement image sets
it.
None of this is reachable: no identity in any existing room can satisfy
`is_member_key`, and the room version cannot be used. Not gated, by design: the
`is_member_key`-guarded raises in `get_domain_from_id`/`get_localpart_from_id`,
which only fire for member-key-shaped strings and turn a 400 "Invalid ID" into a
500. The one remote-reachable path that was a clean 400 before,
`/make_join|leave|knock/{roomId}/{userId}` with a member-key-shaped path
parameter (the handlers compare `get_domain_from_id(user_id)` with the origin),
now validates the parameter as a user ID at the servlet (400 `M_INVALID_PARAM`),
so a malformed request cannot make the Synapse federation client fail over
across every resident on a 5xx. A PDU whose sender is member-key-shaped in a
legacy room already fails at `_check_sigs_on_pdu` and is logged at ERROR with a
traceback on `/send`; only the status code changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add the `experimental_features.msc4428_enabled` flag. When set, `/versions`
advertises `org.matrix.msc4428` and the Rust event serializer adds
`unsigned.org.matrix.msc4428.stable_id` (the sender's user ID) to every
event served to clients, including the events embedded in `unsigned`
(`redacted_because`, bundled edits, thread summaries). The field is subject
to the client's field allowlist like any other unsigned key.
`SerializeEventConfig.__init__` gains a required `msc4428_enabled` argument
(not module API).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This PR implements support for profile updates over Sliding Sync:
https://github.com/matrix-org/matrix-spec-proposals/pull/4262. This pr
may be easier to review as a whole than commit by commit.
This builds on the legacy sync profile updates feature
https://github.com/element-hq/synapse/pull/19556, specifically the
profile updates stream it added.
Submitting for early review to get consensus on implementation. There
are some things we would like to add still, from spec, mainly:
* > Homeservers should only consider a profile field update "accepted"
by a client
> once the client returns with a new /sync request with the next /sync
token,
> NOT just after sending down the profile update. The client may never
receive
> response due to network conditions, or a bug in the client
implementation.
* > When a room enters this subset in this connection for the first
time, all requested
> fields from profiles of users in that room MAY be sent down. This
gives the client
> a base set of information for which future field updates can be
applied on top of.
> The homeserver MAY omit some fields and profiles if it believes that
the client has
> already received them, likewise repeat profiles MAY be sent down based
on homeserver
> implementation.
* > Finally, if the list of fields expands to cover a new field ID,
those fields should
> be sent down for all users that are within the current room subset.
Future incremental
> updates will then include changes to this field.
* Additionally, we would need to implement a lazy loading cache similar
to the legacy sync. (not part of MSC as such)
Depending on review these could either be added to this pr, or to keep
this pr from not growing too much, be added in a follow-up pr, as they
are more enhancement to this base sliding sync profile updates
functionality than a part of the core functionality.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>
Co-authored-by: Olivier 'reivilibre <oliverw@element.io>
MSC3266 is merged in v1.15, let's stabilize it as part of #18731
1. Add support for the stable `/_matrix/client/v1/room_summary/`
endpoint, keeping both unstable endpoints for compat
2. Remove the experimental `msc3266_enabled` flag
Fixes: #8088
Previously we would perform OIDC discovery on startup,
which involves making HTTP requests to the identity provider(s).
If that took a long time, we would block startup.
If that failed, we would crash startup.
This commit:
- makes the loading happen in the background on startup
- makes an error in the 'preload' non-fatal (though it logs at CRITICAL
for visibility)
- adds a templated error page to show on failed redirects (for
unavailable providers), as otherwise you get a JSON response in your
navigator.
- This involves introducing 2 new exception types to mark other
exceptions and keep the error handling fine-grained.
The machinery was already there to load-on-demand the discovery config,
so when the identity provider
comes back up, the discovery is reattempted and login can succeed.
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Part of: MSC4354 whose experimental feature tracking issue is
https://github.com/element-hq/synapse/issues/19409
Follows: #19340 (a necessary bugfix for `/event/` to set this metadata)
Partially supersedes: #18968
This PR implements the first batch of work to support MSC4354 Sticky
Events.
Sticky events are events that have been configured with a finite
'stickiness' duration,
capped to 1 hour per current MSC draft.
Whilst an event is sticky, we provide stronger delivery guarantees for
the event, both to
our clients and to remote homeservers, essentially making it reliable
delivery as long as we
have a functional connection to the client/server and until the
stickiness expires.
This PR merely supports creating sticky events and receiving the sticky
TTL metadata in clients.
It is not suitable for trialling sticky events since none of the other
semantics are implemented.
Contains a temporary SQLite workaround due to a bug in our supported
version enforcement: https://github.com/element-hq/synapse/issues/19452
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
Default values will be 1 room per minute, with a burst count of 10.
It's hard to imagine most users will be affected by this default rate,
but it's intentionally non-invasive in case of bots or other users that
need to create rooms at a large rate.
Server admins might want to down-tune this on their deployments.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
The main goal of this PR is to handle device list changes onto multiple
writers, off the main process, so that we can have logins happening
whilst Synapse is rolling-restarting.
This is quite an intrusive change, so I would advise to review this
commit by commit; I tried to keep the history as clean as possible.
There are a few things to consider:
- the `device_list_key` in stream tokens becomes a
`MultiWriterStreamToken`, which has a few implications in sync and on
the storage layer
- we had a split between `DeviceHandler` and `DeviceWorkerHandler` for
master vs. worker process. I've kept this split, but making it rather
writer vs. non-writer worker, using method overrides for doing
replication calls when needed
- there are a few operations that need to happen on a single worker at a
time. Instead of using cross-worker locks, for now I made them run on
the first writer on the list
---------
Co-authored-by: Eric Eastwood <erice@element.io>
I have a use case where I'd like the Synapse image to start up a
postgres instance that I can use, but don't want to force Synapse to use
postgres as well.
This commit prevents postgres from being started when it has already
been explicitly enabled elsewhere.
* complement: enable dirty runs
* Add changelog
* Set a low connpool limit when running in Complement
Dirty runs can cause many containers to be running concurrently,
which seems to easily exhaust resources on the host. The increased
speedup from dirty runs also seems to use more db connections on
workers, which are misconfigured currently to have
`SUM(workers * cp_max) > max_connections`, causing
```
FATAL: sorry, too many clients already
```
which results in tests failing.
* Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake
* Debug logging
* Only run flakey tests
* Only adjust connection pool limits in worker mode
* Move cp vars to somewhere where they get executed in CI
* Move cp values back to where they actually work
* Debug logging
* Try p=1 to see if this makes worker mode happier
* Remove debug logging
* Fully qualified docker image names for the main Dockerfile and Complement related.
* Fully qualified docker image names for Dockerfiles associated with building Debian release artifacts.
This one is harder and is separate from the other commit in case it wasn't correct or was unwanted. I decided to
do the expansion on the docker images in the Dockerfile itself, instead of the various source places that build
which distribution that is selected, as it would have been more invasive with the scripts breaking up the string
for tagging and such. This one is untested.
* Changelog
* Update docker/Dockerfile-workers
* Update docker/complement/Dockerfile
---------
Co-authored-by: reivilibre <olivier@librepush.net>
* Enable Complement tests for Faster Remote Room Joins on worker-mode
* (dangerous) Add an override to allow Complement to use FRRJ under workers
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Fix race where we didn't send out replication notification
* MORE HACKS
* Fix get_un_partial_stated_rooms_token to take instance_name
* Fix bad merge
* Remove warning
* Correctly advance un_partial_stated_room_stream
* Fix merge
* Add another notify_replication
* Fixups
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Fix presence test
* Newsfile
* Apply suggestions from code review
* Update changelog.d/14752.misc
Co-authored-by: Erik Johnston <erik@matrix.org>
* lint
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true.
Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
