complement main has no tests/msc4429 (matrix-org/complement#849 is
unmerged), so since 19556 merged, every complement CI run on develop
build-fails on the missing package (obscured by a gotestfmt panic on the
mid-stream module downloads). Drop the package here so this PR's CI is
meaningful; restore the line once complement#849 lands.
Removing M_UNKNOWN_POS is about well-formed positions the server no
longer recognises; a malformed token is a plain client error like
everywhere else. Reverts part of 20feb1ef15 and keeps a test pinning
the 400.
- /versions now advertises org.matrix.msc4525, honouring both the global
flag and the per-user experimental feature (as Erik requested inline).
- An unparsable pos no longer 400s: it is treated like an unrecognised
one and the connection starts afresh (MSC4525 has no client error
path).
- The aging lane now also reserves page slots for never-sent rooms, so
continuous traffic in already-delivered rooms can't stall the initial
drain indefinitely.
- total_rooms is counted before partial-state filtering so cold-start
progress reporting isn't understated.
Add MSC4525 to the per-user ExperimentalFeature registry: the servlet is
always registered but 404s unless the feature is enabled globally
(msc4525_enabled) or for the requesting user via the admin
experimental-features API.
Candidates were derived purely from the event stream, so a read receipt
(or room account data change) in an otherwise quiet room was deferred
until someone spoke in it. Rooms with undelivered receipt/account-data
changes - tracked PREVIOUSLY on the per-connection stream maps, or with
activity in the token range - now count as candidates when the relevant
extension is enabled: the room wakes into the page (usually as an empty,
filtered-out entry) and the extension delivers the data.
With room-config tracking off, prev_room_sync_config was always None, so
the lazy-member accounting in get_room_sync_data never ran and incremental
responses were state-deltas-only: a timeline sender's membership that the
connection had never seen was silently omitted. Since required_state is
immutable per connection, treat the previous config as the current one,
which reduces _required_state_changes to exactly the lazy-member
bookkeeping.
The store only needs user/requester/conn_id from the sync config, and is
shared between sliding sync and paginated sync (MSC4525), so type its
methods with a Protocol both configs satisfy instead of SlidingSyncConfig
(fixes the mypy failure on PaginatedSyncConfig at the
get_and_clear_connection_positions call site, and drops the now-redundant
ignore on record_new_state).
- No M_UNKNOWN_POS: an unrecognised pos is treated as absent (nothing
trusted from the token); the connection starts afresh and rooms come
down as never-sent. Clients have no error path.
- required_state is immutable per connection and always taken from the
current request: room configs are no longer persisted or diffed
(track_room_configs flag, off for paginated sync).
- num_live dropped from the wire (derivable: previously-sent rooms only
receive live events, initial rooms are all-historical).
- Extension lists/rooms scoping ignored: an enabled extension applies
to the rooms in the response (PaginatedSyncExtensionHandler).
Tests: unknown-pos-starts-afresh, extensions-without-scoping, num_live
absence; sliding sync suites unaffected (174 still green).
Also gitignore localtest/, the local validation server's scratch dir.
Covers initial-sync paging + drain (pending/total_rooms, most-recent
first, no duplication or loss), per-room gapping (limited + prev_batch,
only the newest limit events), incremental backlog paging, history
semantics for never-sent rooms, and top-level required_state.
New unstable endpoint POST /_matrix/client/unstable/org.matrix.paginated_sync/sync
behind experimental.paginated_sync_enabled. The client sends page_size /
limit / history + a single top-level required_state; the server returns
the changed rooms (most recently active first, at most page_size of
them, at most limit new events each with an explicit per-room gap
beyond), plus pending (rooms that did not fit) and total_rooms.
The implementation is deliberately a thin layer over sliding sync:
PaginatedSyncHandler subclasses SlidingSyncHandler (room data fetch,
extensions, connection store and notifier integration are shared), the
servlet subclasses SlidingSyncRestServlet (room/extension serialisation
shared), and the paging cursor is the existing NEVER/PREVIOUSLY/LIVE
per-connection room tracking - candidates that don't fit the page are
recorded unsent and picked up next request. Fairness: when more rooms
are pending than fit, a quarter of the page is reserved for the rooms
whose undelivered updates are oldest. The expanded-timeline hack is
disabled for this endpoint (history is /messages' job).
Broken out of #20027 because I'd like to have it land first.
> It seems a lot of time in our trial tests goes towards setting up the
database. (The same is probably true of Complement too)
>
> We haven't done a full schema for about 20 schema versions, so no
surprise!
>
> As a result, I want to produce a full schema soon.
When running `make_full_schema.sh`, these drop statements now cause the
error:
```
Parse error near line 2: table event_search_content may not be dropped
Parse error near line 3: table event_search_segments may not be dropped
Parse error near line 4: table event_search_segdir may not be dropped
Parse error near line 5: table event_search_docsize may not be dropped
Parse error near line 6: table event_search_stat may not be dropped
Parse error near line 7: table user_directory_search_content may not be dropped
Parse error near line 8: table user_directory_search_segments may not be dropped
Parse error near line 9: table user_directory_search_segdir may not be dropped
Parse error near line 10: table user_directory_search_docsize may not be dropped
Parse error near line 11: table user_directory_search_stat may not be dropped
```
It seems SQLite has cracked down on code that edits the internal tables.
Because SQLite dumps the schema with `CREATE TABLE IF NOT EXISTS` for
these virtual tables, it's harmless to leave them in the schema dump.
---------
Signed-off-by: Olivier 'reivilibre' <oliverw@matrix.org>
To make it easy to share to people and aid in debugging when they run into
performance issues.
Unfortunately, the Grafana UI doesn't make the export or import steps
easy so it involves some manual Grafana API calls.
As first explored in
https://github.com/element-hq/synapse-rust-apps/pull/397
Without this change, it was linking to `develop` as we were on the
`develop` branch because of the previous `merge-back` step at this point
in the release process.
Follow-up to https://github.com/element-hq/synapse/pull/19984 as this
was an oversight and I assumed we would still be on the `release-v1.158`
branch by that point.
Fix `RemoteJoinHelper` signing events with mismatched room version
compared to the `room_version` arg. The default room version on
`develop` is `11` but the `RemoteJoinHelper` `room_version` arg defaults
to `10` (room version mismatch). This mismatch wasn't present where this
fix was developed
(https://github.com/element-hq/synapse-private/pull/136) as the default
room version was only recently bumped to `11` via
https://github.com/element-hq/synapse/pull/18680 (not even in a release
yet).
Fixes the CI being broken on `develop` :x::
```
[ERROR]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/federation/test_federation_join_upgraded_room.py", line 298, in test_no_transfer_when_tombstone_does_not_match
join_helper.join(local_user_id, local_user_tok)
File "/home/runner/work/synapse/synapse/tests/federation/_remote_join.py", line 350, in join
self._test_case.helper.join(remote_room_id, local_user_id, tok=local_user_tok)
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 195, in join
return self.change_membership(
File "/home/runner/work/synapse/synapse/tests/rest/client/utils.py", line 333, in change_membership
assert channel.code == expect_code, (
builtins.AssertionError: Expected: 200, got: 400, PUT /_matrix/client/r0/rooms/!remote-room:other.example.com/state/m.room.member/@user1:test?access_token=syt_dXNlcjE_JSEarRndiAGqtydnrdLn_33qlLD -> resp: b'{"errcode":"M_UNKNOWN","error":"No create event in state"}'
tests.federation.test_federation_join_upgraded_room.FederationJoinUpgradedRoomTestCase.test_no_transfer_when_tombstone_does_not_match
```
These tests were originally introduced
https://github.com/element-hq/synapse-private/pull/136 (developed
private as this was part of the Synapse security release) and introduced
into the public codebase via
https://github.com/element-hq/synapse/commit/cbc6934821aab314506c5957223c60c74eeda091
This means we can point to a single source of truth instead of
duplicating the content to the tag and GitHub release. Less to manage
and worry about when you make some updates to the changelog (maintenance
burden). We also get to avoid the content sitting in the vendor lock-in
GitHub releases.
We point to
`https://github.com/element-hq/synapse/blob/{repo.active_branch.name}/CHANGES.md`
as it will have the relevant changelog entry at the top and won't change
as we archive releases on `develop`. Even for RC releases after the main
release goes out, the entry will still be towards the top. We could try
to get the heading anchor for the specific section but I thought that it
wasn't necessary (nice but more complex).
Matthew Hodgson