4247 Commits

Include renames

Author	SHA1	Message	Date
Kegan DougalandClaude Opus 5	cc2477541c	Name our own user on a join we send to a member key room ... A membership event carries the claim about who its member key belongs to, and a join that makes no claim leaves its sender attributable to nobody: not routable, not shown to clients, soft failed by every server that receives anything further from them. The joins this server sent to member key rooms made no claim. The gap was in where the linkage information is filled in. For events a client asks us to create, that happens while the event dict is being translated into member keys, which is also where a client's own attempt to assert linkage information is thrown away. A join sent over federation never goes through that path: the template comes from the resident server, we fill in the client's content and sign it, and nothing along the way says who we are. So fill it in there too, once the room version coming back from `/make_join` shows the room wants it: our domain, and the user's localpart unless they are erased, in which case their name is not published again. Whatever the client asked for is dropped first, as it is everywhere else — a client may not say who a member key belongs to, only the server holding it may. The event is then signed by the member key and co-signed by us, which is what makes the claim worth anything to the server receiving it. Also stop recording a link against a member key this server holds. Links describe other servers' keys, and for our own we already know the answer, so the row would be at best redundant. It becomes actively wrong once a member key is moved between servers: the new server's claim about it is still not ours to write down against our own copy. The tests drive the outbound half with a stubbed resident server, so what we put on the wire is checked rather than assumed. The tests for the inbound half build their events by hand and so never exercised this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-16 14:57:00 +02:00
Kegan DougalandClaude Opus 5	cac8b6e343	Learn, verify and use MSC4430 member links ... A member key names no server, so a room full of them is a room whose members cannot be routed to, served to, or shown to anyone. What a key stands for is the linkage information on its `m.room.member` event, and this teaches the server to read it: another server's member can now join one of our member key rooms over federation, and we learn who they are from the event that put them there. A member key's link is the linkage information on its *current* `m.room.member` event, and nothing else. A member who has left has no link, because the event that removed them was somebody else's and asserts nothing about them. Making the mapping a function of the room's current state means every server that has the state agrees on it, no history is needed to compute it, and an event arriving late or by backfill cannot change it. Two signatures decide what a claim is worth. The member key's own makes the claim theirs, and without it the claim is ignored entirely: on an event the member sent, the `sender` signature is already that; on an event somebody else sent about them, it is the signature under `state_key`. The current server signing key of the claimed domain makes the claim *verified*, and that is the half that needs the network, so it is the half that can fail for reasons that are nobody's fault: a domain that does not answer leaves the link unverified rather than refused, and unverified links are re-checked on a small bounded loop so that a member whose server was briefly unreachable does not stay invisible until they next change their membership. What a link is used for depends on which half holds. The claimed domain is where the room's traffic goes, verified or not: an unverified claim is somewhere to send to, and grants nothing. Only a verified link makes a server a member server — able to be told the room's history, to be served its state, to be treated as present in it — because anybody can claim anybody's domain. Events from a key with no verified link are soft failed, so their state changes still count (membership and power levels are decided by keys, not by who holds them) but their messages are not shown to clients. Joining a room of this kind works from either end. A joining server does not know the room's version until `/make_join` answers, so it cannot know whether to name its user by ID or by key: it sends the user ID in the path and the key in `org.matrix.msc4430.member_key`, which a server that has never heard of MSC4430 ignores. The key is held in memory until the room turns out to want it, so joining a room without member keys leaves nothing behind, and it is stored before the join is sent, so a join that succeeds is never one we cannot sign for afterwards. A resident server accepts a key in either place and refuses a bare user ID for such a room. Because the sender of a membership event is a key and not a user ID, the check that the sender belongs to the requesting server no longer applies: the origin is simply who relayed it, and what the member claims about their own server is in the event, signed by them. The end-to-end tests build the remote half by hand, so that a server can be made to vouch for its member, decline to, vouch for a different one, or lie about somebody else's. They cover a verified join, an unverified one, a re-check that succeeds on a later attempt, a member key that moves to another server and takes its member with it, and a moderator who bans a member while naming a domain they control and changes nothing about who that member is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-16 14:02:14 +02:00
Kegan DougalandClaude Opus 5	c320ced0fc	Only a member key may assert its own linkage information ... An `m.room.member` event's `content.localpart` and `content.domain` claim who the member key in `state_key` belongs to, and the server named by `content.domain` signs the event to vouch for the claim. That signature is not enough on its own: a moderator kicking or banning a member can name a domain they control, vouch for the claim with that domain's signing key, and produce a verifiable but false statement about somebody else — defacing an innocent member with an abusive localpart, or pointing them at a server that then receives their traffic. So require the claim to come from the member too. The member key named by `state_key` signs the event under the entity `state_key`, with the member keys' constant key ID, and linkage information on an event the member did not send is ignored unless it is there. A moderator cannot produce that signature without the member's key, and the absence of one costs them nothing: the kick or the ban still takes effect, since this is not a validity rule. It only decides whether anyone believes what the event says about who was kicked. The events this server builds follow suit. A membership event sent by one of our users about another — an invite — now carries the target's own signature alongside ours, which is what makes it worth anything to the servers that receive it. A kick, a ban or an unban carries no linkage information at all: we would be asserting something we are not entitled to assert, and the target's localpart has no business being published on somebody else's event. Members' own events are unchanged; their `sender` signature is already their assertion, and a second copy of it would say nothing more. This replaces the previous arrangement, where the server's signature vouching for linkage information was made under the constant `state_key` rather than under the server's own name. That distinguished a claim about the target from one about the sender, but it did not establish who was entitled to make it. The entity name goes back to the server's name, as everywhere else in Matrix, and `state_key` now names what it says: the member key in the event's state key, signing for itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-15 17:12:45 +02:00
Kegan DougalandClaude Opus 5	c692133ed4	Co-sign MSC4430 linkage information under the state_key entity ... An `m.room.member` event in a member key room carries two signatures: the member key named by `sender` signs the event, and the server named by `content.domain` co-signs the claim that the key in `state_key` belongs to `content.localpart` there. That co-signature was made under the signing server's own name, which leaves a reader unable to tell what is being asserted: an invite is sent by one member key and co-signed by the *invitee's* server, so a signature by `example.com` on it could plausibly stand behind either the sender or the target. Name the entity for the member the claim is about instead. The constant `state_key` joins the existing constant `sender`, and a signature under it is, by construction, an assertion about the member named by the event's `state_key` whoever sent the event. The key ID stays the signing server's real key ID, since that is what names the published key a verifier must check it against; only member keys use the constant `ed25519:1`. A signature under a server's name is now refused outright rather than treated as linkage information, so the only entities an event in such a room may carry are `sender`, `state_key` and the authorising member key of a restricted join. Nothing yet verifies a co-signature, so this changes only the events this server produces, and only in member key rooms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-15 16:05:01 +02:00
Kegan Dougal	b342193742	Register and deregister a user's event stream against a room by their user ID rather than the membership event's state key, which in an MSC4430 member key room is the member's key and matches no stream — plus the missing deregistration on leave.	2026-10-15 15:15:45 +02:00
Kegan DougalandClaude Fable 5.1	104a1baa9c	C4: Moderate and upgrade MSC4430 member key rooms ... Kicks, bans and unbans, power levels and room upgrades in a member key room, through the client API, on one homeserver. A moderator can now act on any member, including one whose key this server holds nothing for (as federation will bring them), and a room can be upgraded into a member key version and from one member key version to another. Rooms without member keys are untouched. Naming a member. Clients name a membership target on `/invite`, `/kick`, `/ban`, `/unban` and `PUT /state/m.room.member/{target}` by user ID, as ever, or in a member key room by key. `MemberKeyHandler.parse_membership_target` turns what the client sent into a `RoomMemberIdentity` (a `UserID` or a `MemberKey`): a user ID is a user; a key we hold is the user behind it, so that everything the membership code does for a user of ours (suspension, invite rules, profiles, notifications, the user-keyed membership tables) applies, and the user ID becomes the key again when the event is built; any other key, one nobody vouches for or one with a verified link to a member of another server, is a `MemberKey`, which `update_membership` accepts for a kick, a ban or an unban and refuses for anything else with 400 (there is nobody to invite, join or knock as), and which the event then names as it is. `update_membership_locked` and `_local_membership_update` take the `MemberKey` through the leave and ban paths and skip what only a user has: an account to be suspended, a profile, an invite to look up, a device to wake, a "left room" to record. A `third_party_signed` block (a third-party invite to exchange on a join) is refused with 400 in these rooms whatever the target: exchanging it builds a membership event from the user IDs the client sent. A `MemberKey` is never ours (`hs.is_mine`), so a kick of an invited key goes the way a kick of a remote invitee does. `kick_guest_users`, which found our guests by the domain of their state keys and so skipped every member of such a room, finds them through the keys we hold. The event the kick or ban produces is sent by the moderator's key and signed by it alone: `translate_event_dict` lets a membership of `leave` or `ban` whose sender is not its target name any key, and attaches no linkage information for a key we do not hold (we vouch for nothing about it), while a target we do hold gets ours and our co-signature as before. A member may still name their own key on the state API, which is handled as themselves. Power levels. The `users` of an `m.room.power_levels` event are translated the same way as membership targets: a user ID of ours becomes the user's key in the room (a user with no key yet is given one, so that a power level can be granted ahead of a join, as in any room), a key is kept as it is (so any member can be named by key), a user ID of another server is refused with 400, naming the same member twice (by user ID and by key) is refused with 400 rather than letting one entry win, and anything else is left for the auth rules to reject. This covers the power levels a room is created with (`power_level_content_override`) as well as later events. Third-party invites bind a user ID to a membership, which means nothing for keys: `do_3pid_invite` refuses them in these rooms with 400, before any identity server is contacted, and so does an `m.room.third_party_invite` state event. Upgrades. `/upgrade` into a member key version works, from a room with member keys or without one. The members a clone carries over (the power levels' `users`, the bans, and with `auto_member` the joins and the pending invites) are named in the old room as it knows them, which in a member key room are keys that mean nothing in another room; `clone_existing_room` and `_update_upgraded_room_pls` carry them by the users behind them (`_users_behind_room_identities`, `_power_levels_by_user`), so that building the new room's events mints new keys for them, and the creator removal that MSC4289 versions need is applied to the user-keyed map. Before anything is created, and once the requester is known to be allowed to send the tombstone (the refusal names members of the room), `_check_upgrade_member_identities` refuses an upgrade of a member key room to a version without member keys (400, `M_UNSUPPORTED_ROOM_VERSION`), and an upgrade into a member key version whose old room's power levels or bans name a member who is not one of our users (400, naming them), since the new room could not hold them; the old room is left as it was, with no tombstone. (The check and the clone read the room's state separately; a power level or ban naming such a member that lands between the two is refused when the clone builds the event, after the new room's create event exists, as any upgrade that fails midway.) A joined or invited member that maps to no user is left behind with a log line rather than refusing the upgrade: there is nobody to join or invite. `additional_creators` on `/upgrade` into these versions are validated as user IDs (they become keys in the create event). Two things the clone needed: the signing helper and `translate_event_dict` accept the frozen content of a copied event (a `Mapping`, not a `dict`; translation copies it before editing), and `create_and_send_new_client_events` chains `prev_state_events` through a batch in an MSC4242 state DAG room, each state event taking the previous batched state event as its parent, so that a join sent in the same batch as its invite is authorised against it (the auth check already looked for parents within the batch; the builder did not produce them). And the rejections `auto_member` sends into the old room for the invites it re-sends are, in a state DAG room, sent after the tombstone rather than during the clone: the tombstone is built before the clone and sent after it, so whatever the clone sends into the old room is the tombstone's sibling, and in a state DAG room the merge of a rejection with the tombstone's branch, where the invite still stands, is left to state resolution with nothing to order the two by (the rejected invite came back as an invite on some runs). `clone_existing_room` returns those events for `_upgrade_room` to send once the tombstone is in. Rooms without a state DAG take the previous path in both places. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:39 +02:00
Kegan DougalandClaude Fable 5.1	6e7672f595	C3: Serve MSC4430 member key rooms to clients ... What a client sees of a member key room: every event names a member key as its sender and, through MSC4428, as its stable ID; `member_info` resolves the keys to users; the requester is recognised by their key wherever the server decides what they may see or which events are theirs; and `/members` and `/joined_members` are keyed by key. One homeserver only, but the code answers for members of other servers too (a verified link names the user it claims, anything else is nobody). `member_info`. MSC4428's map, which only named erased users (the stable ID of an ordinary room is the user ID and the entry is omitted), now gives every member key an entry: the user behind a key we hold, the user named by a verified link for another server's key, and `null` for an erased user or a key nobody vouches for (an unverified claim is anyone's to make). The resolution is one batched store lookup (`get_user_ids_for_identities`, exposed by the member key handler as `user_ids_for_identities`), so the map costs rooms without member keys nothing. The map already covers `/sync` (full and incremental), sliding sync, `/members` and `/joined_members`, which are keyed by state key and so by key; nothing there changes. An erasure is recorded on the change stream under the stable ID that changed, which in these rooms is the member's key, so the entry that flips to `null` is the one clients hold. Visibility. `filter_and_transform_events_for_client` decided what the requester may see from their membership under their user ID, which the state of a member key room does not hold: a joined user saw nothing. It now looks the requester up under their key in each member key room the batch touches, for both the allow check and the membership the event is annotated with, and maps senders back to users, per room (a key is only a user in the room it was looked up in), for the two per-user checks: erased senders, whose events are pruned for users who joined later, so a key with no attributable user is not erased; and ignored senders, where a member may be ignored as the user behind their key or as the key itself, since a key with no attributable user is the only handle a client has on it. `filter_event_for_clients_with_state`, which push evaluation uses to decide who may see an event, does the same per local user, for the ignore lists as for the memberships, including for the out-of-band membership events it returns to their target. The push evaluator itself resolves the sender (so a user is not notified of their own events), the invitee and the inviter (so an invite notifies its target, and the invitee's invite rules see the inviter as a user, where one is attributable). The requester's own events. The serializer echoes `transaction_id` and `delay_id` only to the sender's own session, by comparing the sender with the requester's user ID, which in a member key room never matches. `SerializeEventConfig.requester_member_keys` (a new keyword-only argument, `None` by default, so existing constructions are unchanged) carries the requester's keys and the Rust serializer accepts a sender among them; `create_config` fills it from the cached per-user keys when the feature is on. Keys are unique across rooms, so a key of the requester's can only ever be their own sender. Sync. Where the handlers compared the requester's user ID with a sender or a membership state key, they now use the identity the room knows them by (`room_identity_for_user`: their key where they have one, their user ID otherwise, or the cached map of their keys per room): the incremental sync reads the requester's previous membership under their key, so a change to their own membership (a display name) is an ordinary timeline event rather than the room sent again as newly joined, and a leave is newly left; the heroes of a nameless room no longer include the requester; the lazy-loading full sync fetches the requester's own membership under their key; a leave the user sent themselves is told from a kick by their keys per room, in sync and in the sliding sync room lists; the join-race guard asks the user-keyed membership table; and sliding sync's `$ME` is the requester's key, including in the computation of which required state has changed between requests. An invite's sender is resolved to a user (where attributable) for the ignore list and the invite rules, in sync and in both sliding sync room lists: a key on the ignore list hides the invite too, and an unattributable key no longer reaches the invite rules, which parse user IDs and would have failed the whole sync. The members handed to presence and to device lists, by sync, by the device handler for newly joined and left rooms, and by the room initial sync, are mapped to users first, so clients are told about users' devices and presence, not keys'. The three storage queries that find a user's membership changes in `current_state_delta_stream` and `room_memberships` share one clause that, with the feature on, also matches the keys we hold for the user, each scoped to its room (keys are public strings anyone can name as a state key elsewhere) and spelled out from the cached per-user keys so that every arm of the clause can use the tables' indexes; a user with no keys, and the feature being off, get the legacy clause byte for byte. Sliding sync's "newly joined" and "newly left" detection, and the device handler's shared-room tracking, run on those queries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:39 +02:00
Kegan DougalandClaude Fable 5.1	b1228f6d9b	C2: Create, join, send and leave in MSC4430 member key rooms ... With `msc4430_enabled` on, the member key room version is registered and a user can create such a room, join it, send events, change their own membership, invite another user of this server and leave, all by user ID as today, while every event in the room names member keys and is signed by them. One homeserver only; members of other servers, moderation (kick, ban, power levels), restricted joins and what clients are served back (`/sync`, `/event`, `/members`) are outside this change. Where a user ID becomes a key. Clients speak in user IDs, so the translation happens once, in `EventCreationHandler.create_event`, before the event is built: `MemberKeyHandler.translate_event_dict` replaces the `sender`, the `state_key` of an `m.room.member` event and the `additional_creators` of a create event with the users' member keys in the room, creating a key for a user who has none (a key never changes, so a user who leaves and re-joins keeps theirs). A membership event about one of our users gets our linkage information (`content.localpart` and `content.domain`); whatever the client put there is dropped first, since linkage information is a server's assertion, and an erased user gets no `localpart`, so their name is not published again. A user ID of another server, as sender, target or additional creator, is refused (403 or 400) rather than quietly sent as a user ID the room cannot hold, and so is a member key the client chose itself as a target or additional creator: a key nobody vouches for would be a seat in the room for whoever holds its private half (the membership servlets still parse user IDs, so a key as a target is refused there too). A room with a remote invitee is refused before anything is created, not by the invite after the room exists, and a room cannot be upgraded into this version (the clone names user IDs in power levels, bans and invites). The handler is also where the rest of the server asks what an identity means: `local_user_id_for_identity` (ours, by user ID or by a key we hold in that room), `user_id_for_identity` (also the verified link of another server's key), `domain_for_identity` (routing only: our name for our keys, the claimed domain of a linked key) and `room_identity_for_user` (how to look a user of ours up in a room's state: their key where they have one, their user ID otherwise, which the state then does not contain, as it should not). Three other places compared the sender with a user ID and now resolve it: a suspended user redacting their own event, the admin-redaction rate limit, and room participation. The create event. In these room versions the room ID is derived from the create event, and the create event names and is signed by the creator's key, so the key has to exist before the room ID does. `translate_event_dict` generates such keys as "pending" (an expiring in-memory cache), they sign the create event from there, and once the event is built `create_event` stores them under the new room ID; if building fails they are dropped at once. A pending key is ours only for the create event itself (no room yet): in any existing room it is nobody's, so a key minted for one room cannot act in another while it waits. A user named twice by one create event (as creator and additional creator) gets one key; a create event sent into an existing room is refused before any key is made; and the room ID collision retry starts each attempt from the content the client sent, not the translated one. Room creation then looks the creator's membership up by their key in the state map it batches the rest of the initial state against, and refuses the version when the flag is off even though the registry (process wide) may know it, which is also why the tests can run flag-off cases. `default_room_version` cannot name this version: the server config validates it before the experimental config registers it, so rooms are created with an explicit `room_version`. Signatures. `EventBuilder.build` asks `MemberKeyHandler.get_signers_for_event_dict` who signs and `create_local_event_from_event_dict` signs with exactly those: the sender's member key under the entity `sender`, if we hold it; our server signing key on an `m.room.member` event whose target is a key we hold and whose linkage information is exactly what we vouch for about that user (our domain, their localpart unless they are erased, in which case none), which is what lets other servers verify the member's link; and the authorising member's key on a restricted join that names one of ours. An event whose sender is not a key we hold gets no signature at all (it is only hashed), so the server never signs as a key it does not own; the co-signature is decided from the store, never from the dict, so no caller can have us vouch for a name that is not the user's; and the signing helper refuses any signer the event may not carry, so a wrong choice here could not produce one either. A third-party rules module that replaces an event is as untrusted as a client: its dict is translated again, and it may not change who authorised a join. The builder's validation accepts canonical member keys as sender and membership target in these rooms, and nothing else. Membership handling. `update_membership_locked` looks the target and the requester up in the room's state under their room identity (their key once they have one), so re-joins, duplicate events and the ban checks see the right membership; the previous membership of a new event is read under the event's own state key; whether this server is in the room is answered from the members that map to our users rather than by domain; a pending invite is read by user ID (that table is keyed by user) and the inviter's server is found through `domain_for_identity`, so a join via a remote inviter would still route; the join notification maps the joining key to its user; and an invite in such a room is refused with 400 unless the invitee is a key we hold. Hosts in a room. Registering the version makes the "which servers are in this room" paths reachable for the first time, and they are what the federation sender consults for every event. They took the domain of every member; a member key has none, and on Postgres the SQL that extracts domains from state keys returned a `None` host for such a room. `get_hosts_for_identities` on the member keys store now gives the server behind each identity (a user ID's domain, our own name for keys we hold, the claimed domain of a key we have a link for, nothing for the rest), and `_get_joined_hosts`, `get_current_hosts_in_room` and `get_current_hosts_in_room_ordered` use it for member key rooms (the ordered form is unordered there: nothing in the state keys to sort by). The sender itself decided whether an event was ours by the sender's domain, which for a key said "not ours" and dropped every event of such a room before reaching any of this; it now asks the identity map, and finds the server of a rescinded invite's target the same way rather than by parsing the key, which would have raised outside the guard that protects the sending loop. A member key room holds only our own members, so its hosts are this server and nothing is sent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:39 +02:00
Kegan DougalandClaude Fable 5.1	6c5fccd908	C1: Persist MSC4430 member key membership keyed by user where the server needs a user ... In an MSC4430 member key room the membership state keys and senders are per-room Ed25519 keys, not user IDs. The room's own tables keep those keys (`current_state_events`, `room_memberships`, the state and auth chains), but several tables and caches are keyed by *user* and are what everything downstream of persistence consults to answer "which rooms is this user in": `local_current_membership`, the sliding sync membership snapshots, `event_txn_id_device_id` (a foreign key to `devices`), `get_rooms_for_user` and its relatives, and the membership stream cache that gates incremental sync. This change fills those with the user behind each key we hold, on the main process and on workers, and leaves rooms without member keys on exactly the path they took before. The member key room version is not registered, so none of it is reachable. The local identity map. Storage cannot import the handler that owns identity resolution for the rest of the server, so the storage half lives on the member keys store: `MemberKeysWorkerStore.get_local_identity_map_txn(txn, room_id, identities)` and its async counterpart `get_local_identity_map`. Both turn room member identities into local user IDs: our users' IDs map to themselves, member keys we hold *in that room* map to their user, and everything else (other servers' users, other members' keys, and our own keys for other rooms) is dropped. Member keys are only looked up when `msc4430_enabled` is on; off, a member-key-shaped state key is as malformed as it always was and is dropped as `is_mine_id` dropped it. The lookup is one batched query, and none at all when no identity is a member key, so rooms without member keys cost nothing. The map is built once per persisted batch at the top of `_persist_events_txn` from every sender, every membership state key and every membership change in the state delta, and threaded to `_persist_transaction_ids_txn`, `_store_room_members_txn` and `_update_current_state_txn`; the only identities not known until inside the transaction are the room's members when we are leaving it, which `_update_current_state_txn` adds with a second lookup. The partial-state resync path (`update_current_state`) builds its own map for its delta. The async form serves `_calculate_sliding_sync_table_changes` and the persistence controller, from the cached per-key lookups; keys never change once created, so a cached answer can only be missing, never wrong. What is keyed by user now. `_update_current_state_txn` writes `local_current_membership` rows for the mapped users; the sliding sync snapshot rows carry the mapped user (and the member key as `sender`, which is what the room says); `_store_room_members_txn` does the same for out-of-band memberships (remote invites and their rejections); `_persist_transaction_ids_txn` stores the mapped sender, and `get_already_persisted_events` (the persistence-time deduplication of a transaction ID) looks it up the same way, so two racing sends of one transaction ID in a member key room still collapse to one event; a sender the map does not know is stored as before, so the foreign key makes a key that fails to map fail loudly rather than silently drop the transaction ID; the persisted-events metric counts events from our keys as local. Caches and workers. Membership cache invalidations are keyed by the event's state key, in `_invalidate_caches_for_event` (every event, and the events stream for workers) and in `_invalidate_state_caches` (current state changes, and the caches stream). In a member key room that key is the member key, so the user-keyed caches `_invalidate_caches_for_event` touches (`get_rooms_for_user`, `_get_rooms_for_local_user_where_membership_is_inner`, `get_sliding_sync_rooms_for_user_from_membership_snapshots`, `get_invited_rooms_for_local_user`, `did_forget`, `get_forgotten_rooms_for_user` and the membership stream cache) are now also invalidated for the user behind the key: once per membership event at the event's own position, next to the per-event invalidation, and once from `_update_current_state_txn` at the batch's position for members with no event in the batch (a state reset, or everyone in the room when we leave it). Workers get these as a row of their own on the caches stream (`MEMBER_KEY_USERS_CACHE_NAME`, keys `[room_id, stream_ordering, *user_ids]`, handled by `_invalidate_user_caches_for_member_keys`): the events stream rows name the key and a worker cannot map it, and the current-state rows would record the change at the caches stream's own position, which the membership stream cache ignores when it is below the worker's starting event position and misplaces when it is above. Only users whose state key is a key get these rows; a user whose state key is their user ID has already had exactly these invalidations, so rooms without member keys see no extra rows. `_invalidate_state_caches` itself no longer parses member keys as user IDs: a key names no host, so a membership change under one drops the `is_host_joined` / `is_host_invited` entries for the whole room, which needs those two caches to be tree caches keyed by room first. That is the one change not gated on member keys: the caches change shape for every room (an entry can be dropped by its room prefix), the alternative being to flush them for every room on every change in a member key room. The replication client's notifier wake-up on workers parsed the membership state key as a user ID, which for a key raised and aborted the whole batch of replication rows (nothing in that batch, in any room, woke a sync on that worker); it now wakes the user behind one of our keys, and nobody for anyone else's. Reads. `get_rooms_for_user` and `get_rooms_for_users` union `local_current_membership` for local users when the flag is on; it is keyed by user ID in every room and agrees with `current_state_events` in rooms without member keys, so the union adds only the member key rooms. With the flag off the legacy query runs byte for byte. `get_membership_changes_for_user` also matches the user's member keys (`room_memberships.user_id` is the state key) when the flag is on, scoped to each event's room: keys are public strings, so a membership event in room B naming the user's room A key is not a membership change of theirs. In the persistence controller, extremity pruning groups senders by origin to drop an old extremity sooner when the same origin sends again; in a member key room a key is its own origin (the stricter reading), and the "never drop our own events" walk uses the map rather than `is_mine_id`. `_is_server_still_joined` maps the delta's member keys to users and passes user IDs to `is_local_host_in_room_ignoring_users`, which queries `local_current_membership`; before this the server would not have noticed its last member leaving a member key room. Remote member keys. Those reach `handle_potentially_left_users_txn` as opaque identities (nothing in the map), where "do we still share a room with them" is answered from `current_state_events` by state key and the unsubscribe is a no-op delete plus one cache invalidation. Run against the unchanged code, the tests show how an unadapted site fails: thirteen of the fourteen persisting tests end in `MemberKeyHasNoDomainError` (500) from `_invalidate_state_caches`, the first site on every path to parse a member key as a user ID, which hides the ones behind it (`_update_current_state_txn`, `_calculate_sliding_sync_table_changes`, `_persist_transaction_ids_txn`, `_prune_extremities`, `_is_server_still_joined`, the replication client, all adapted here). The one silent failure was the out-of-band path in `_store_room_members_txn`: `is_mine_id` answered False for the invitee's key and the invite was simply never recorded for the user. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:39 +02:00
Kegan DougalandClaude Fable 5.1	e01562f1e0	B3: Verify and authorise events in MSC4430 member key rooms ... In a room whose members are MSC4430 member keys, the `sender` of every event and the `state_key` of every membership event is a per-room Ed25519 key, and the event is signed by that key rather than by the sender's server. This change teaches the two places every received event must pass, signature verification and the auth rules, to work on keys. The member key room version is not registered, so none of it is reachable; it is the acceptance boundary every server has to agree on, so it depends on nothing but the event. Signatures (`_check_sigs_on_pdu`): in a member key room the event must carry a valid signature under the entity `sender` by the key decoded from its `sender`, with key ID `ed25519:1`, over the redacted event as for server signatures; a restricted join must additionally carry a valid signature by the member key named in `join_authorised_via_users_server`, under that member key string. No server signature is required and no key is fetched, so every server reaches the same verdict. The sender and the authoriser are parsed strictly: a non-canonical base64 encoding of a key is not that key (otherwise one public key would have two identities), and a user ID is rejected. A server co-signature on a membership event's linkage information is not checked here, on purpose: the MSC makes it decide whether the member's link is verified, not whether the event is accepted, and some membership events legitimately carry no linkage information at all. Hash checking, redaction on a hash mismatch, the policy server and the spam checker are unchanged. Auth rules (`event_auth`): `validate_event_for_room_version` checks that the sender is a canonical member key and has signed the event, that a membership event's `state_key` is a canonical member key, and that a restricted join names a canonical member key as authoriser and carries its signature. The `m.federate` domain comparison is skipped in member key rooms, in both the general and the membership rule: keys have no domain, so it cannot be evaluated, and in such rooms `m.federate: false` means no federation at all. Power level `users` and `additional_creators` must be member keys; `check_valid_additional_creators` therefore takes the room version, and the room upgrade servlet passes the new one. Third-party invites need no rule change: the rule compares `signed.mxid` with the `state_key` and verifies `signed` with the key published in the inviter's `m.room.third_party_invite` event, so an identity server that signs the member key produces an acceptable invite; how it learns the key is unspecified. Everything else in the rules already works on opaque identities: membership lookups, power levels, creator power and the restricted-join authoriser are all compared as strings. A membership event with no `membership` in its content is handled without raising on both new paths (the legacy paths have the same latent `KeyError`). Redactions: in room versions since v3 a redaction by a sender without redact power is applied only if both senders are from the same server, a check deferred until the redacted event is known and made in three places (the event cache, the read path, and persistence of an existing redaction). Those three now share `redaction_sender_matches`, which keeps the server comparison for every other room version and, in member key rooms where senders have no server, applies only when the two senders are the very same key. That is stricter than before (a server's other users can no longer redact a member's events without power), and the only rule that needs no network. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan DougalandClaude Fable 5.1	0f3e69f9c1	B2: Add storage for MSC4430 member keys and member links ... MSC4430 identifies room members by a per-user, per-room Ed25519 "member key" rather than a user ID, and lets a member's `m.room.member` event carry the user ID it claims to be (`content.localpart`, `content.domain`), co-signed by the claimed domain's server signing key. This adds the two tables and the store behind that, with no callers: nothing in the server creates keys or learns links yet, and the member key room version cannot be used. `msc4430_member_keys` holds the keys we hold for our own users, one per (local user, room), with the private seed. `MemberKeysWorkerStore.create_local_member_key` is insert-if-absent and returns whichever key is stored, so a user's key in a room can never change once it exists, however many concurrent callers try (the earlier monolithic attempt raised `IntegrityError` here). The store derives the public form from the signing key it is given, so the two columns cannot disagree, and rejects any key ID other than `ed25519:1`. Every lookup by key takes the room the key was seen in: a key of ours from room A appearing in room B's events does not resolve there (the earlier attempt mapped keys without a room and gave users phantom memberships). Lookups are cached per (room, user), per (room, key), per user and per room, and `create_local_member_key` invalidates all four on the cache stream, so workers see new keys. `msc4430_member_links` holds what we have learned about keys we do not hold: the claimed domain (routing information, kept whether verified or not), the claimed localpart (NULL once the event that carried it is redacted, after which the key maps to no user), and whether the link is verified. A verified link always names the `m.room.member` event and the server signing key ID that proved it; the `MemberLink` type and a CHECK constraint both enforce this, so "verified" means exactly one thing. Our own keys get no link row: a link is a claim someone else made, and the keys we hold are already known to be ours. `upsert_member_link` is a plain replace that returns the previously stored link, so a caller can see what changed (the mapped user, the domain); which events may update a link is the caller's policy, not the store's, and `get_member_link_txn` plus `upsert_member_link_txn` let it read, decide and write in one transaction. `mark_links_unverified(domain, key_id)` downgrades every link a rotated or withdrawn server key vouched for, in one bulk cache invalidation, keeping the event and key ID so the links can be found again (`get_links_for_signing_key`) if the events are re-signed; a key ID alone does not identify a key, so re-checking a link means re-checking the co-signature on its event against the domain's current key material, which the schema says. `get_member_keys_for_domain_in_room(verified_only=...)` separates routing (claimed) from trust (verified), and answers for our own server name from the keys we hold, never from links: a link claiming our domain for a key we do not hold is somebody else's claim. `get_member_links_for_claimed_user` is the reverse mapping; it returns every claimant (verified first) rather than picking one, because an unverified claim is anyone's to make and that lookup is how a user ID in an invite, kick or power level change becomes a key. `clear_link_localpart_for_redacted_event_txn` is the erasure hook (it clears every link learned from the redacted event). Purging a room clears its links; the keys we hold deliberately survive a purge, since a user's key in a room must not change if the room is re-joined. The stored seed cannot be revoked (MSC4430 has no member key rotation; only rotating the server key helps), and the schema says so. The store is mixed into `RoomMemberWorkerStore` (so it is present on workers) rather than registered on `DataStore`, because the membership queries are the code that will need it. `LocalMemberKey` and `MemberLink` are frozen attrs types in `synapse/storage/member_keys.py`, next to the other storage return types; `SigningKey` is added to the mypy plugin's immutable types so a `LocalMemberKey` can be a cached value. `synapse_port_db` learns the boolean column. `MemberKeyHandler.is_local_identity` (and so `hs.is_mine_identity`) now answers for member keys by looking the key up in its room, instead of raising. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan DougalandClaude Fable 5.1	5161187a50	B1: Define the MSC4430 room version, the MemberKey type and the signing helpers ... Rust: `RoomVersion.msc4430_member_keys` and `MSC4430V12` ("org.matrix.msc4430.12" = `MSC4242V12` + the flag); redaction of `m.room.member` keeps `content.domain` and drops `content.localpart` in such rooms, so the reference hash, the event ID and the member key signature cover the claimed domain but not the claimed localpart; `membership_field::{DOMAIN,LOCALPART}`; `org.matrix.msc4430` in `/versions`; `.pyi` updated. The version is defined but *not* registered in `KNOWN_ROOM_VERSIONS`, not even when the flag is on. Python: `MemberKey` in `synapse.types` next to the other identifier types, with the grammar `^@[A-Za-z0-9_-]{43}$` (anything with a `:` is a user ID), `is_member_key` (syntactic, regex only, for hot paths) and `MemberKey.from_string` (strict: rejects non-canonical base64, so each public key has exactly one member key string). `MemberKeys` constants (`SIGNATURE_ENTITY = "sender"`, key ID `ed25519:1`) and `EventContentFields.MSC4430_{LOCALPART,DOMAIN}`. `supports_msc4430_member_keys` is a `TypeIs[MSC4430Event]` modelled on `supports_msc4242_state_dag`; `MSC4430Event` subclasses `MSC4242Event`, since every member key room is a state DAG room, so one predicate narrows to both. Signing: `add_hashes_and_signatures_from_signers(room_version, event_dict, signers, *, server_name)` hashes once and then applies `(entity, key)` signers in order; `compute_member_key_room_signature(room_version, event_dict, signer, *, server_name)` adds one signature to an already hashed event (the authoriser's signature on `/send_join`, our co-signature on an inbound `/invite`) without rehashing a PDU we have not checked. The three signer kinds are: the sender's member key under `sender`; our own server name co-signing linkage information; the authorising member key of a restricted join under the member key string itself. Together these are the one choke point for signing in member key rooms, and they refuse with `MemberKeySigningRefusedError` (500) any signer the MSC does not allow: a `sender` signature by a key other than the one `sender` names, a key ID other than `ed25519:1`, a server co-signature under any name but ours or on anything but an `m.room.member` whose `content.domain` is us, or an authorising signature on anything but a join that names that key in `join_authorised_via_users_server`. A bug in choosing signers therefore cannot make us sign as a key we do not hold or vouch for linkage information that is not ours. An empty signer list hashes only (`/make_join` templates). Loudness: `get_domain_from_id` and `get_localpart_from_id` raise `MemberKeyHasNoDomainError` (a `SynapseError`, 500: it is a server bug) when given a member key, on the no-colon branch only, so user IDs and aliases are untouched. Because the HTTP layer logs every `SynapseError` at INFO without a traceback, and the error may be caught before it gets there, both MSC4430 errors log themselves at ERROR with the stack of the raise site: the point of them is to name the unadapted code path. `hs.is_mine_id` is left alone: it is called on raw client strings (state keys, sliding-sync `required_state`, `/users/{id}/report`, module API) and on remote-controlled room content (`/upgrade` moving a canonical alias), where "False if malformed" is relied upon and a member-key-shaped string in a legacy room is just malformed. `hs.is_mine_identity(room_id, identity)` is the sanctioned question for room identities; it answers for user IDs and raises `MemberKeyHasNoDomainError` for member keys. The Prometheus counter `synapse_msc4430_unsupported_path_total{server_name,path}` and `record_unsupported_path(server_name, path)` (counts always, warns once per path per process) are for branches that deliberately skip member keys. The identity module is `synapse/handlers/member_keys.py` (`MemberKeyHandler`, `hs.get_member_key_handler()`). The `MemberKey` type and `is_member_key` live in `synapse.types` because `get_domain_from_id` needs them and the module cannot import handlers. Config: `msc4430_enabled` implies `msc4428_enabled`; the Complement image sets it. None of this is reachable: no identity in any existing room can satisfy `is_member_key`, and the room version cannot be used. Not gated, by design: the `is_member_key`-guarded raises in `get_domain_from_id`/`get_localpart_from_id`, which only fire for member-key-shaped strings and turn a 400 "Invalid ID" into a 500. The one remote-reachable path that was a clean 400 before, `/make_join|leave|knock/{roomId}/{userId}` with a member-key-shaped path parameter (the handlers compare `get_domain_from_id(user_id)` with the origin), now validates the parameter as a user ID at the servlet (400 `M_INVALID_PARAM`), so a malformed request cannot make the Synapse federation client fail over across every resident on a 5xx. A PDU whose sender is member-key-shaped in a legacy room already fails at `_check_sigs_on_pdu` and is logged at ERROR with a traceback on `/send`; only the status code changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan DougalandClaude Fable 5.1	61da72e7f5	A3: Add a stream for MSC4428 member_info changes with no event anchor ... Changes to a member's `member_info` entry that no event in the room anchors (erasure, and MSC4430 link verification) were only delivered when the member next appeared in a response. Give them a stream so that syncing clients are woken and told. * `StreamToken.member_info_key`: a trailing `MultiWriterStreamToken` component (`StreamKeyType.MEMBER_INFO`); tokens without it still parse. * `msc4428_member_info_changes` (delta 95/01) behind a multi-writer id generator with no fixed writer list, as for cache invalidations: any worker may write (erasure runs wherever deactivation is handled). `MemberInfoChangesStream` replicates rows; workers wake room members. * Retention: rows older than 30 days are pruned hourly on the background worker and the prune point recorded; a since token older than that is sent the full map per room instead of deltas. * Producer: `MemberInfoHandler.on_user_erased`, from `deactivate_account`, one row per room the user has any membership in. * Consumers: sync v2 reads the changes for all joined rooms once per sync and includes a room with an empty timeline when it has any; sliding sync includes such rooms when choosing what to send and resends the full map for rooms it last sent at an older position. The stream exists always, like every stream, but nothing writes to it unless `msc4428_enabled` is set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan DougalandClaude Fable 5.1	9a26683b32	A2: Add the MSC4428 member_info map, anchored on events ... Add `org.matrix.msc4428.member_info` to `/sync` (joined and left rooms), sliding sync, `/members`, and `user_id` to `/joined_members` entries. A new `MemberInfoHandler` computes the map; entries are only produced where the mapping differs from the stable ID itself, so it only names erased users (`user_id: null`, via `are_users_erased`). A room's map covers the stable IDs the response mentions: senders of its timeline, state and sticky events, targets of its membership events, and the room's heroes. On an initial sync without lazy-loaded members that is every member; a lazy-loading client gets the entries for members it loads later from `/members`, which always carries the map. On sync the map is omitted when empty, and `member_info` counts towards whether a room is included in the response. Changes with no event anchor (the erasure itself) are delivered by the next event from or about that member, or on the next full sync; a stream for them is A3. With `msc4428_enabled` off the responses are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan DougalandClaude Fable 5.1	45cfb3bc93	A1: Advertise MSC4428 and add unsigned.org.matrix.msc4428.stable_id ... Add the `experimental_features.msc4428_enabled` flag. When set, `/versions` advertises `org.matrix.msc4428` and the Rust event serializer adds `unsigned.org.matrix.msc4428.stable_id` (the sender's user ID) to every event served to clients, including the events embedded in `unsigned` (`redacted_because`, bundled edits, thread summaries). The field is subject to the client's field allowlist like any other unsigned key. `SerializeEventConfig.__init__` gains a required `msc4428_enabled` argument (not module API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 09:22:38 +02:00
Kegan Dougal	0aea320a6e	Linting	2026-10-02 16:48:46 +01:00
Kegan Dougal	f086c2043a	Merge branch 'kegan/4242-inbound' into kegan/4242-inbound-events	2026-10-02 16:46:47 +01:00
Kegan Dougal	b65f0b7025	Review comments	2026-10-02 16:38:34 +01:00
Kegan Dougal	947b9107e0	Various bug fixes and enable Complement tests for MSC4242 ... - Enable MSC4242 complement tests - Fix an issue which caused cascade rejections to not be applied to events in the /send_join response becau se we didn't remember in-memory which events were rejected. - Accept rejected events in the /send_join response, it gives us a bit of robustness at no safety cost. - Recalculate room state if backfilled events caused the state DAG fwd extrems to change. - Don't filter out already seen events from event_map, so the statement "the event_map is the complete stat e DAG" still holds. - Add validation to length of prev_state_events - Don't validate auth_events length for state DAG events as it is done too early, before we've had time to calculate the auth events (and validation is moot anyway given it isn't attacker controlled anymore) - Let backfilled events update state dag fwd extrems - Add test for edge case with backfilled state events	2026-10-02 16:38:34 +01:00
Kegan Dougal	0e579e2823	Calculate auth events on events and auth based on them	2026-10-02 16:38:34 +01:00
Kegan Dougal	326f040798	Add is_state_dag_connected to check the events returned in /send_join	2026-10-02 16:38:34 +01:00
3 people	4f5524043c	Use beautifulsoup4 instead of lxml for URL previews (#19301) ... Use `beautifulsoup4` instead of `lxml` for URL previews. This offers some nicer APIs when parsing HTML and avoids using `libxml`, [which is unmaintained](https://gitlab.gnome.org/GNOME/libxml2/-/commit/9c80a89af2fdf4f853892f84e46580f4902658ba). I haven’t done a full regression against commonly previewed sites, but I expect this will give similar (or better) results. beautiulsoup also handles decoding the charset for us, which is less custom code. --------- Co-authored-by: Andrew Morgan <andrew@amorgan.xyz> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2026-10-02 15:46:34 +01:00
Erik Johnston	5a8c4b3990	Fix bug where /sync would not return membership changes due to lazy-loaded membership cache (#20289) ... In `/sync` there is a mode called lazy loaded (LL) members. In this mode `/sync` does not return all memberships in the room, instead it only returns a) membership changes, and b) the memberships of senders of events in the timeline (if not previously sent). Synapse implements a lazy-loaded member cache to avoid repeatedly sending down the same memberships when a single user sends multiple events. It does this by storing for each user/device the memberships it has previously returned in an LRU cache. Note that it is fine for the server to return the same membership, clients handle that fine, but it is a waste of bandwidth. The current lazy-loaded membership cache has three intertwined flaws: 1. It does not handle retries or clients re-using old tokens. 2. It incorrectly applies the membership cache to membership changes, not just memberships added by lazy-loading. 3. It does not correctly handle multiple "sync streams" from the same device. In this PR we explicitly fix the first two issues in two commits, and the third issue is mitigated: 1. Alongside each membership in the LL cache we store the `next_batch` position of the response that carried it. A client that processed a response syncs from its `next_batch` next. When a request arrives with a `since` token, we delete every membership recorded after that token, as they were sent in responses the client never processed. Memberships recorded at or before `since` are kept. 2. We always return memberships that have changed, even if they appear in the LL member cache. These changes also half mitigate the issue of running multiple `/sync` loops per-device (though we do not see this in the wild), as we'd at least always return the changed membership states. Fixes #19978 and #20278, supersedes #20279	2026-10-02 09:48:14 +01:00
Andrew Morgan	b3c67110a5	Bump Twisted in poetry.lock from 25.5.0 to 26.4.0 (#20259)	2026-09-29 17:50:55 +00:00
Andrew Morgan	c9b61fc2a3	Stop real threadpool for test DB pool before replacing it with threadless test implementation (#20272)	2026-09-29 16:25:32 +00:00
Olivier 'reivilibreandEric Eastwood	d1874cdc37	Prevent MSC4354 Sticky Events being sticky when they were redacted prior to persistence. (#20232) ... Part of: MSC4354, Part of: https://github.com/element-hq/synapse/issues/19641 This PR persists events in redacted form when they already have valid redactions. That's not directly a sticky events-specific change (and I feel it makes sense to redact immediately before persistence when we already have the redaction). This causes a nice effect which is that the event's stickiness is lost immediately before persistence, because the `msc4354_sticky` field is redacted by the redaction algorithm. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>	2026-09-29 15:22:34 +01:00
Paul ChobertandOlivier 'reivilibre	db618e276d	Fix appservice ephemeral read receipts being permanently skipped on large receipt bursts (#20108) ... Fixes https://github.com/element-hq/synapse/issues/20096. Part of https://github.com/element-hq/synapse/issues/18118 The spec requires read receipts to be delivered to interested application services ([Pushing ephemeral data](https://spec.matrix.org/v1.19/application-service-api/#pushing-ephemeral-data)): > If the `receive_ephemeral` settings is enabled in the registration file, homeservers MUST send ephemeral data that is relevant to the application service via the transaction API, using the `ephemeral` property of the request's body. > > `m.receipt`: MUST be sent to the application service under the same rules as regular events, meaning that the application service must have registered interest in the room itself, or in a user that is in the room. Ephemeral data delivery to application services was added in Matrix v1.13 from [MSC2409](https://github.com/matrix-org/matrix-spec-proposals/pull/2409) (see the [v1.13 changelog](https://spec.matrix.org/v1.19/changelog/v1.13/#application-service-api)). ## Problem When more than 100 read receipts are covered by a single receipt stream update, only the last 100 are sent to appservices. The rest is permanently skipped. ## Proposed Fix - Looping over the receipts in batches of 100 items and updating the cursor on every iteration - Keep the fast-forward behavior for new appservices as intended in https://github.com/matrix-org/synapse/pull/8744 --------- Co-authored-by: Olivier 'reivilibre <oliverw@element.io>	2026-09-29 12:06:15 +01:00
Ankit Jha	18c10b8075	Fix 500 error when user-interactive auth requests send a null or non-object auth (#20274) ... Fixes #15871 ### What was broken Sending `{"auth": null}` to any endpoint that requires user-interactive auth (for example `POST /keys/device_signing/upload`, `DELETE /devices/{deviceId}` or `POST /register`) returned a 500 instead of the usual 401 with the available flows. A non-object `auth` on endpoints that do not validate the body with a model (such as `/register`) could also 500, e.g. `{"auth": ["session"]}`. ### Root cause `AuthHandler.check_ui_auth` did `clientdict.pop("auth", {})`, so the `{}` default only applies when the key is missing. An explicit `null` came through as `None` and the following `"session" in authdict` raised `TypeError`. `get_session_id` had the same assumption that `auth` is a dict. ### Pull Request Checklist * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Signed-off-by: Ankit Jha <ankit.jha@tradomate.one>	2026-09-28 18:36:39 +00:00
Ankit Jha	929e3524d1	fix: reject user creation via the admin API when delegating to MAS (#20241) ... Signed-off-by: Ankit Jha <jhaankit373@gmail.com> Signed-off-by: Ankit Jha <ankit.jha@tradomate.one>	2026-09-28 10:17:37 +00:00
Erik Johnston	7630492e90	Add metrics for wait_for_stream_token timeouts per lagging stream (#20095) ... Add a metric for the number of times we see a stall due to being given a future token. This would have caught https://github.com/element-hq/synapse/issues/20080, where lots of sync streams got stuck due to a stream not getting replicated correctly.	2026-09-25 12:09:38 +01:00
Johannes MarbachandAndrew Morgan	b8bd6f93a1	Add push rules for MSC4075: MatrixRTC invites and notifications (#20227) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2026-09-25 10:40:24 +00:00
3 people	e863ce7fab	Fire deferreds from Rust futures without taking the GIL (#20252) ... Tokio tasks currently complete by taking the GIL and calling `reactor.callFromThread`. Taking the GIL may block for a period of time, and we do not want that to happen on the tokio reactor threads (as that can block other work from happening). To avoid this, we instead add a work queue that we can add to from Rust without taking the GIL, which is drained by the reactor. We signal to the Twisted reactor that it should wake up by using a unix socket pair, which is exactly how `reactor.callFromThread` works. --- The self-pipe trick is basically where you create a pair of unix sockets connected to each other. One end is added to the reactor so the reactor is woken up when there are bytes to read, and when another thread needs to wake up the reactor it just needs to write a byte into the other unix socket. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2026-09-25 09:49:14 +01:00
Jason Robinson	b0d44f9352	Fixes to the profile update stream when a user leaves a room (#20203) ... This pull request fixes a few issues with the profile update stream, when a user leaves a room. This is basically just mimicking what we already had for when someone leaves a room, and they no longer share any rooms, but in reverse - ie when we leave a room, and no longer share rooms with some users. This was missed in the implementation when adding the profile update stream for legacy and sliding sync. [Fix missing profile update stream rows on user leaving room](https://github.com/element-hq/synapse/commit/e953322226731a99718ebe13139b9c5f05ae4b62) When a user leaves a room, profile update stream rows are generated with the `LEFT_ROOM` action for each user in the room that no longer shares a room with the user who left the room. This also needs to happen in reverse. The user who left the room needs to have a profile update stream row for each user they no longer share a room with. If we don't do this, clients may keep stale data around even after they don't share a room with a user, which may mean they don't know when to refetch profiles after re-joining a room with the stale profile data user. [Clear out old profile update stream rows when leaving a room](https://github.com/element-hq/synapse/commit/e7642bf141b098c1eefaf74655e5e7e18415500b) When we leave a room, ensure profile update stream rows are cleared out for every user we no longer share a room with. This is the same as what happens when someone else leaves a room, but in reverse. The only remaining profile update stream row should be the `LEFT_ROOM` action. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2026-09-24 21:22:08 +00:00
Will Hunt	b29c3c6764	Omit og:image entirely if the media is quarantined when handling /preview_url (#20211) ... This is to prevent the case where URL previews return a MXC that immediately 404s because the media in question has been quarantined. This is mostly to help implementations which currently show an ugly empty preview due to the MXC being sent down, despite being invalid. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [ ] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2026-09-24 17:48:07 +00:00
Olivier 'reivilibre	62cb420a0e	Send all of a room's MSC4354 Sticky Events down oldschool /sync when a user joins it. (#20233) ... Part of: MSC4354 whose experimental feature tracking issue is https://github.com/element-hq/synapse/issues/19409 Part of: #19662 --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2026-09-24 15:28:06 +01:00
3 people	64f0590705	MSC4140: cancel a user's delayed events when their account is deactivated (#20247) ... [MSC4140](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/4140-delayed-events-futures.md), "Account deactivation": > when an account is deactivated, the homeserver MUST cancel that account's delayed events which have not yet been added to a room's event DAG. These cancelled records MAY be removed immediately, including their stored event content, as an exception to the usual finalised-record retention policy. Deactivation currently leaves the user's scheduled delayed events, including their content, in place, and they are still attempted at their scheduled time: | Delayed event scheduled by the user | What happens after deactivation today | | --- | --- | | Message or state event, fires after deactivation has made the user leave the room | The send fails with a 403 (user not in room) and the record is dropped | | Message or state event, fires before deactivation has made the user leave the room (leaving happens room by room in the background) | The event is sent | | `m.room.member` join for themselves in a public room | The deactivated user re-joins the room | ### What changes - Deactivating an account (client or admin API) removes the user's unsent delayed events as its first step, and re-arms the send timer for whatever is scheduled next. - Deactivation can run on a worker, while the send timer lives on the main process, so the cancellation goes through a new replication request. - A delayed event whose send has already started is left alone, as with a normal cancel: it is already on its way into the DAG, and the send path removes its record itself. Suspension and locking are unchanged. ### After #19038 Today a cancelled delayed event is simply deleted, so this PR deletes the user's records too. #19038 changes cancellation to keep the record and mark it as cancelled ("finalised"), so that clients can look up what happened to a delayed event. Once it has landed, deactivation should probably finalise the user's records as cancelled the same way, rather than delete them. The MSC allows either: it permits removing the records immediately, content included, as an exception to the usual retention of finalised records, which is worth doing at least when the user asks to be erased. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Andrew Ferrazzutti <af_0_af@hotmail.com> Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>	2026-09-24 10:05:33 +02:00
Paul ChobertandOlivier 'reivilibre'	c0b7224e85	Reject limit_profile_requests_to_users_who_share_rooms without require_auth_for_profile_requests (#20231) ... As I was working on https://github.com/element-hq/synapse/pull/20218, I noticed what seemed to be an illegal configuration of synapse. - `require_auth_for_profile_requests`: blocks profile requests unless authenticated - `limit_profile_requests_to_users_who_share_rooms`: blocks profile requests unless authenticated user share a room with requested user This, I think, should be an illegal config: ``` require_auth_for_profile_requests = false limit_profile_requests_to_users_who_share_rooms = true ``` As of now, with such a config the shared-room check is never applied: a profile can be requested anonymously, and also by an authenticated user who doesn't share a room. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>	2026-09-23 16:33:17 +00:00
3 people	fbfa7bcfd1	Fix push badge count ignoring a room's notifications when another room's summary is up to date (#20237) ... The count of unread notifications is computed in two phases: 1. From the summaries (`event_push_summary`) when those are up to date with the user's last read receipt 2. Otherwise from counting the push actions (`event_push_actions`) after that receipt The problem in this process is that the threads whose summary is up to date are identified with only their `thread_id`. But all main timelines share the same `"main"` thread_id, so as soon as one room has an up-to-date summary, phase 2 skips the main timeline of every room. The fix identifies the up-to-date summaries with their `(room_id, thread_id)` pair. Authored by @sandhose. Found while investigating the `TestThreadedReceipts` Complement flake (#15517, #18537), but it is a bug on its own. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Quentin Gliech <quentingliech@gmail.com> Co-authored-by: Devon Hudson <devonhudson@librem.one>	2026-09-23 16:08:43 +00:00
Erik Johnston	818f2b25bb	Fix error log when writing a large response. (#20246) ... We sometimes see a lot of `ERROR` logs like the following: ``` Closing scope Scope<... master.write_bytes_to_request> which is not the currently-active one None ``` This error is generated by `opentracing.Scope` checking that it is the "active" one. Synapse tracks "active" spans via logcontexts, so this indirectly asserts that the scope is closed in the context it was opened in. However, the producer methods are often called from the reactor and therefore withing the sentinel logcontext, which produces the error above. This specifically happens when the producer tries to write large responses but gets paused, and then later resumes. The fix is to simply use `Span` directly, rather than scopes. `Span` does not perform the checks. I noticed this when deploying #19979, though it is unrelated.	2026-09-23 09:46:50 +01:00
NEVIL ANSON DSOUZAandDevon Hudson	1d7880b9cb	Make task scheduler concurrency configurable (#18308) (#20230) ... ### Summary Resolves #18308. Makes the `TaskScheduler`'s maximum concurrent running tasks configurable via `task_scheduler.max_concurrent_tasks` in the homeserver configuration and defaults it to `2` (reduced from the previous hardcoded limit of `5`). ### Motivation Twisted's `adbapi.ConnectionPool` defaults to 5 database connections (`cp_max=5`). When the `TaskScheduler` previously executed up to 5 tasks concurrently, it could exhaust the entire database connection pool, starving regular API and synchronization requests on smaller homeserver instances. Lowering the default to 2 preserves at least 3 connections for foreground requests while allowing concurrent tasks to progress, and gives administrators the ability to tune the limit according to their host and connection pool sizing. ### Changes - Added `TaskSchedulerConfig` (`task_scheduler.max_concurrent_tasks`) with validation (must be a positive integer) and registered it in `HomeServerConfig`. - Updated `TaskScheduler` to enforce the configured concurrency limit and updated class default to 2. - Updated config documentation and JSON schema. - Added config tests in `tests/config/test_task_scheduler_config.py` and updated unit tests in `tests/util/test_task_scheduler.py`. - Added newsfragment in `changelog.d/18308.feature`. --------- Signed-off-by: nevil06 <nevilansondsouza@gmail.com> Co-authored-by: Devon Hudson <devonhudson@librem.one>	2026-09-22 21:33:59 +00:00
Eric Eastwood	625b331768	MSC4311: Use full PDU's in stripped state (like invite_room_state) over federation and always include m.room.create event (#19723) ... ### Background This PR was originally just trying to remove the flawed [MSC4311](https://github.com/matrix-org/matrix-spec-proposals/pull/4311) partial implementation as client side API's like `/sync` should still use stripped events. But it turns out we were just re-using the client logic for the federation side and things might break if we didn't include the full `m.room.create` event so this PR now introduces MSC4311 support to use full PDU's in the `invite_room_state`/`knock_room_state` in the federation API's. The flawed implementation was originally introduced in https://github.com/element-hq/synapse/commit/0eb7252a230811e59679cc7e55b92dac26532efc (no PR I assume because part of Hydra security fix) which was part of [Synapse v1.136.0](https://github.com/element-hq/synapse/blob/7530874a1250d6ad975b39582a784c594d29a505/CHANGES.md#synapse-11360-2025-08-12). Spawning from reviewing https://github.com/element-hq/synapse/pull/19722 and noticing that we have [`TestMSC4311FullCreateEventOnStrippedState`](https://github.com/matrix-org/complement/blob/1e2e12eebc1edb27bbf12108ec849a8254b6ddcd/tests/v12_test.go#L1341-L1376) in Complement which already passes even though that test looks [flawed](https://github.com/matrix-org/complement/pull/791#discussion_r3132468346): > I think this test is mixing up what [MSC4311](https://github.com/matrix-org/matrix-spec-proposals/pull/4311) proposes. Perhaps these were changes to the MSC that came after? > > For the client API's like `/sync`, it only proposes that `m.room.create` is a required *stripped* state event. > > For the federation API's, alongside requiring `m.room.create`, it also mandates using the full event PDU format for all events in the `invite_room_state`/`knock_room_state` on `m.room.member` events (in `unsigned`) ### What does this PR do? 1. Always use stripped state for client API's 1. Remove flawed [MSC4311](https://github.com/matrix-org/matrix-spec-proposals/pull/4311) partial implementation (as explained above) 1. Sanitize stripped state when we receive events over federation 1. Use full PDU's when sending `invite_room_state`/`knock_room_state` over federation 1. Validate PDU's and warn when receiving `invite_room_state`/`knock_room_state` over federation 1. In the future, we will strictly validate and reject Complement tests: https://github.com/matrix-org/complement/pull/796 --- Part of https://github.com/element-hq/synapse/issues/19414	2026-09-22 11:57:54 -05:00
Erik Johnston	f3ae564877	Report each ID generator's current position as a metric (#20097) ... When a stream advances in the database but stops being replicated to a process, that process's view of the stream freezes. Requests that wait for it to catch up to a token issued by another worker then time out and return empty responses indefinitely (see #20080), and nothing exported said so. Report `get_current_token` from every ID generator, on every process. The value is comparable between processes, so a stream that has stopped reaching one of them shows up as divergence with no client traffic needed. It is also the position that `wait_for_stream_token` waits on, so its divergence is the failure itself rather than a proxy for it. Being a watermark over gapless runs of persisted IDs, it also catches a single writer of a sharded stream going quiet, which a maximum across writers would hide behind the writers still being replicated.	2026-09-22 13:16:35 +01:00
Erik JohnstonandClaude Opus 5	9b5697d378	Move per-homeserver Rust state into a RustRuntime object on the HomeServer (#20011) ... Previously the tokio runtime was stashed in a hidden attribute on the reactor object, installed lazily by whichever Rust code first needed it, and started via `callWhenRunning`. Instead, we create a `RustRuntime` (accessible via `HomeServer.get_rust_runtime()`) that holds any per-reactor Rust state, such as the tokio runtime. It is constructed lazily on use. Rust consumers (`HttpClient`, `VersionsHandler`, the Python DB pool wrapper) now receive the runtime or reactor handle explicitly, and the `reactor.run()` / manual-startup workarounds in tests are no longer needed. We also add helper wrappers in Rust for `Reactor` and `HomeServer` that exposes the needed functionality. The aim is to allow us to have a Rust-side clock (mainly to get the current time), that respects the unit test per-reactor time management. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-21 18:48:37 +01:00
Kegan Dougal	8ae810bbc4	Adjust test comments and make created rooms more accurate	2026-09-21 14:46:04 +01:00
Kegan Dougal	b32e9a3605	Cap prev_state_events and the state DAG walk	2026-09-21 14:46:04 +01:00
Kegan Dougal	2d0d65e1d4	Test receiving MSC4242 state DAG events over federation	2026-09-21 14:46:04 +01:00
Kegan Dougal	658523546c	Fetch missing state DAG events via /get_missing_events	2026-09-21 14:46:04 +01:00
Kegan Dougal	67fc7516c7	Calculate auth events on events and auth based on them	2026-09-21 14:46:02 +01:00
Kegan Dougal	d7febaf6cf	Add is_state_dag_connected to check the events returned in /send_join	2026-09-21 14:46:02 +01:00
Erik Johnston	218fc91210	Add a cache to state res to cache more cases (#20185) ... This is an attempt to better cache the cases where there are a large number of extremities to resolve over, which keep slightly changing. This spawns from seeing issues on matrix.org. We already have a cache over the exact state groups being resolved. However, we can do better by caching the inputs into state res (i.e. the conflicted sets), which are more likely to be constant across repeated state res in a room. We key this cache based on a sha256 hash, on the assumption that this will never conflict. Also includes a commit that removes needless copying of the state.	2026-09-21 13:40:31 +01:00