The test only checked that a non-member does not get the announcer's
device change, so it would also pass if the change produced no device
list update at all. A user sharing a regular room with the announcer now
gets that same change in the same test.
A sender removed before anything was announced gets the synthetic join,
then the room is left out of their lists until the first announcement.
It was dropped from the rooms checked for updates outside the lists, so
the connection kept it as sent up to its last token: when the
announcement listed it again, only that announcement went down and any
change made in between, such as a new room name, was lost.
The rooms left out are now kept among the rooms checked for updates,
without being listed. A change while they are left out marks them as
only partly sent, and they come back with everything since. This reuses
the existing tracking of rooms outside the list ranges, instead of
resending the room from scratch when it is listed again, which would
need its own bookkeeping.
Nobody is banned from an announcement room in practice: senders are
removed from the config, which kicks them. Only a leave now gives way to
the synthetic join again.
An announcement room the user is not a member of is now listed when it
holds an announcement, or when the user left, was kicked or was banned
within the request's window, and otherwise left out. A user removed
before anything was announced no longer gets the room on a new
connection, and a user leaving on their own before any announcement gets
the synthetic join in the same response instead of their leave.
The room is then never sent with the user's real leave, so resending it
from scratch afterwards is no longer needed.
A user banned from an announcement room kept their real ban, so they
stopped getting announcements. As in legacy sync, a ban now gives way to
the synthetic join like a kick does, and is left out of `required_state`
and the timeline.
A member removed from the configured senders before anything was
announced got their kick, and the room only came back as joined with the
first announcement. Their room now gets the synthetic join straight away,
at the position of the kick until there is an announcement.
Only kicks count: both paths always keep them in the user's memberships,
while the new tables leave out the user's own past leaves.
Also test that a removed member added back gets their real join and the
power levels that let them post again.
A member who left or was removed from an announcement room first got
their leave, then the synthetic join on the next sync. They now get the
synthetic join instead of their leave in the same sync, as in legacy
sync, once the room has had an announcement. Before that, the leave
still goes down and the room is sent again from scratch later.
A timeline reaching back to the user's leave or kick ended with that
membership after the synthetic join in `required_state`, so clients saw
the user as having left. The user's own membership events are now dropped
from the timeline of a server announcement room, as they already are from
its `required_state`.
A user who left or was kicked while their connection was open never got
the synthetic join afterwards, since only state changes were sent. When a
response sends the user's new leave in an announcement room, the room is
now recorded as never sent, so its next delivery holds the full
`required_state` with the synthetic join.
A user removed from the configured senders is kicked, and both paths
kept the kick as their membership, so they never got the synthetic join.
Only a leave that isn't newly left now gives way to the synthetic join.
On an existing connection, the synthetic join is still not sent after
the kick, as `test_removed_member_existing_connection` shows.
The fallback path kept the self-leave in the membership map, so the
announcement room was skipped and then filtered out of the lists. Only
memberships the lists keep now prevent the synthetic join.
Also test that the room stays in the list, is ordered like the user's
other rooms, and can be subscribed to directly.
This includes the following:
- Implements ['Implicit Forward
Pagination'](https://github.com/matrix-org/matrix-spec/issues/2378).
Remove the extension's `since` and `next_batch` fields. The position is
now tracked via the global `pos`, with per-room connection state
recording how far each room has been sent so that `limit` paginates
without holding `pos` back and lost responses are re-delivered.
- Discussed in
https://github.com/matrix-org/matrix-spec-proposals/pull/4480#discussion_r4165887827
- Honour the `lists` and `rooms` fields.
- Send all unexpired sticky events when a room enters scope for the
first time or is newly joined, and catch up on events sent while a room
was out of scope when it re-enters.
- Enabling the extension partway through a connection returns all
unexpired sticky events for in-scope rooms, and re-enabling it after
disabling it returns the unexpired sticky events sent in the meantime.
I tried to split this up into standalone commits though the bulk of the
change is in the third commit.
---------
Co-authored-by: Olivier 'reivilibre <git.oo@emunest.net>
## What the MSC says
From MSC4140's [Auth rule
evaluation](https://github.com/matrix-org/matrix-spec-proposals/blob/b7cb09664fed684388ba656459d08cfaf786c1d1/proposals/4140-delayed-events-futures.md#auth-rule-evaluation):
> The server SHOULD also check the event could be sent when it is
scheduled and reject the request if not. In particular, servers SHOULD
check the sender is in the room (or otherwise authorized to send the
event) to avoid storing events for rooms the user is not in.
## This PR
- Checks the event against the room's auth rules as if it were sent now,
and returns the same 403 `M_FORBIDDEN` that `/send` and `/state` would.
- Changes to the sender's own membership are exempt: a join or knock can
be sent from outside the room, which is how this PR reads the MSC's "or
otherwise authorized". They are checked by the auth rules when they are
sent.
Here are a few examples:
| Delayed event | Before | After |
| ----------------------------------------------------------------- |
--------------------------------------- |
---------------------------------------- |
| `m.room.message` from outside the room | 200 `delay_id`, fails at send
time | 403 `M_FORBIDDEN` |
| Kick of another member from outside the room | 200 `delay_id`, fails
at send time | 403 `M_FORBIDDEN` |
| `m.room.name` from a member whose power level is too low | 200
`delay_id`, fails at send time | 403 `M_FORBIDDEN` |
| **Exception**: join of a public room from outside it | 200 `delay_id`,
joins at send time | unchanged |
| Join from outside the room, then a message due after it | both
accepted, message sent once joined | join accepted, message 403
`M_FORBIDDEN` |
Part of: MSC4354 (Sticky Events, Experimental Feature tracking issue
#19409)
When a user is the sender of a sticky event that comes down sync, that
user is now in scope for having their lazy-loaded membership sent down
in sync.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Part of: MSC4354 (Experimental Feature tracking issue #19409)
Follows: #20165 (Sticky Event federation backlog + catch-up
transactions)
The MSC says that we need to tell a newly-joining server about all of
our sticky events (yes, even the ones before the server joined the
room).
This PR introduces the mechanism, building on the 'Sticky Event backlog'
mechanism introduced in #20165, to make that happen.
Most of the logic is taken from the original Sticky Events PR #18968,
but the Sticky Event backlog system did not exist back then.
(The benefit of having the persistent Sticky Event backlog is that we
don't need to rely on the federation senders not crashing to make sure
we get around to sending the Sticky Events out.)
- Add NEW_SERVER_JOINED replication command. (This is a cross-worker
'wake-up call' to check for a sticky event backlog)
We send it when persisting a state change that causes a new server to be
in the room.
We handle it by waking up federation senders for the destination,
including checking the sticky event backlog tables.
- When a server newly joins a room, we insert a sticky event backlog row
for that server, if there are any Sticky Events needing to be sent.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Depends on: https://github.com/matrix-org/complement/pull/914
`displayname` and `avatar_url` already did not permit nulls, seemingly
from #8223.
The specification at v1.15 does not permit `null` for those fields
either, so this seems consistent with historical specification.
This only changes behaviour for 'custom' fields.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Follows: #18375
This bugfix is a prerequisite for MSC4354 (Sticky Events, Experimental
Feature tracking issue #19409) changes.
- Add a failing test for join->join being newly_joined
- Fix the join->join transition being treated as a new join
(when prev_events didn't coincidentally mask the problem)
Rationale:
`event_edges` / `prev_events` does NOT link a membership event
to its previous state.
(It could even be someone else's membership!)
In most real-world cases, this would generate a `prev_membership` of
NULL
for every normal membership change.
It appears as though the author intended some kind of 'previous state'
relationship, but if you fix the first branch of the `UNION` query
to use `current_state_delta_stream` for this (which is the source of
state transitions), then it looks very redundant with the second
branch.
It looks like the real reason we want to use the snapshots table is for
out-of-band memberships (outlier memberships).
So we continue using that for those, but ONLY for those.
The `SUSPICIOUS` if block is removed because it appears to have been
papering over this issue by deduplicating the spurious `NULL` ->
non-null transitions,
but when you remove those duplicates at the source, it causes more
trouble
because it doesn't know that what it's 'deduplicating' is actually the
same membership transition.
This means if you join, then out-of-band leave (e.g. last member on the
server
to leave), that room would appear in `newly_joined` on a new initial
sync.
(Tests did in fact catch this, too.)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Closes#20116
> [!NOTE]
> This PR Follows #20115 which already implemented this for
`m.device_list_update`
> User IDs with localparts containing characters outside the range
U+0021 to U+007E, or with
> an empty localpart, are considered non-compliant. For current room
versions, servers must
> still accept events using such user IDs over federation; however they
SHOULD NOT forward
> such user IDs to clients when referenced outside the context of an
event. For example,
> device list updates from non-compliant user IDs would be dropped by
the receiving server.
>
> — [Spec
1.19](https://spec.matrix.org/v1.19/appendices/#historical-user-ids)
This PR prevents from forwarding non compliant user ids for the
following events:
- `m.signing_key_update` 8c3c35e95d
- `m.presence` 26a8dfe862
- `m.typing` 1110cf2d73
- `m.receipt` 9e2da8b0a6
- `m.direct_to_device` 9541232a30
These things are not needed now that MSC2965 has stabilised. In any
case, they
should never have been exposed without a config flag in the first place.
This speeds up the /createRoom calls a tiny bit by not checking the
senders push rules.
This helps speed up a lot of test runs in Element Web which
runs a lot of tests in one-person rooms (and creates a heck of a lot of rooms)
Paul Chobert