Part of: MSC4354 (Experimental Feature tracking issue #19409)
Follows: #20312 (which plumbs `received_ts` into the `Event` struct,
allowing us to calculate this correctly and sanely)
- When writing to `sticky_events`, we now use the `received_ts` stamped
onto the event to calculate the expiry,
rather than taking a second clock reading.
- When emitting the `unsigned.sticky_duration_remaining_ms` (currently
called `unsigned.msc4354_sticky_duration_ttl_ms`) field,
we now also use the `received_ts` as a clamp on the start of the
stickiness,
causing us to calculate the same expiry timestamp as the one saved to
`sticky_events`
(**This is a bugfix, as previously we weren't using `received_ts` here
like the MSC requires.**)
- The calculation of the expiry is centralised into the `Event` struct,
which I hope you'll agree is a
huge sanity improvement.
- We now systematically prevent spammy or rejected events from counting
as sticky.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erikj@element.io>
In a similar vein to #20315, this attempts to speed up Synapse when
delayed events aren't in use. Here we skip doing any work if the room
event stream has not advanced.
Motivated by work on MSC4354, because the sticky events MSC specs that
`received_ts` is used to calculate the
start of an event's stickiness. (It would therefore be convenient to
have it available as part of event metadata, so an event can expose a
cheap helper method to check if it is sticky.)
This commit adds `received_ts` into the `event.internal_metadata` struct,
alongside `stream_ordering`.
I conceptually treat it as a sibling to `stream_ordering` and populate
it in all the same places where `stream_ordering` is populated.
Because this needs `received_ts` to be added to the HTTP replication
response for event persistence,
I also add a temporary compatibility measure that keeps rolling worker
upgrades working during this
transition.
I intend to remove the compatibility code in the next release.
I have manually tested that
```
SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_POSTGRES_HOST=localhost SYNAPSE_POSTGRES_PASSWORD=postgres poetry run trial -j 6 tests.replication.test_sharded_event_persister
```
fails only when both the compatibility code is removed and the
`"received_ts"` key is commented out in the response (to omit it).
The compatibility code causes the test to pass even when `"received_ts"`
is commented out.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Fixes: #19661
Part of: MSC4354
First introduced some helpers for querying a sharded stream, which were moved to
https://github.com/element-hq/synapse/pull/20108.
- SQL builder for 'give me rows between two sharded tokens'
- a function to return the 'new' from_token after a limited read
Then make the sticky events portions of both oldschool and sliding sync use
sharded tokens.
This prevents a 'torn sync' problem where, because the events and sticky
events streams are both sharded with the same writers:
1. persisting a sticky event leads to an event row and a `sticky_events`
row being created for event `e` (on writer `W`).
2. Suppose another writer is still persisting, so the position of the
stream for writer W advances, but the global
['linear'](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html?highlight=%E2%80%9Clinear%E2%80%9D#current-stream-id)
(I have been calling this 'baseline') position does not advance yet.
3. sync is sharding-aware for the regular events stream, so `e` appears
as a real-time timeline event, thus coming down sync
4. since sticky events were not sharding-aware, `e` is not detected as a
sticky event in this sync response. The sticky event position doesn't
advance in the sync token, either.
5. (assume the position advances because other writers have finished
writing and so the 'linear' position advances for the sticky events
stream)
6. the client then syncs again, using the new sync token. This time, `e`
appears in the sticky event section.
If we make sticky events sharding-aware in sync, then `e` would appear
for consideration in both the timeline section **and** the sticky event
section in the first sync.
It then gets deduplicated out of the sticky section since it is
appearing in the timeline section in the same sync.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
Part of: MSC4354 (Sticky Events)
Our locally-sent Sticky Events need to be reliably (within bounds of
possibility) sent to remote homeservers.
This adds the mechanism to make this happen.
Notably omits catching up historical sticky events to homeservers that only just
joined the room.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
In `/sync` there is a mode called lazy loaded (LL) members. In this mode
`/sync` does not return all memberships in the room, instead it only
returns a) membership changes, and b) the memberships of senders of
events in the timeline (if not previously sent).
Synapse implements a lazy-loaded member cache to avoid repeatedly
sending down the same memberships when a single user sends multiple
events. It does this by storing for each user/device the memberships it
has previously returned in an LRU cache. Note that it is fine for the
server to return the same membership, clients handle that fine, but it
is a waste of bandwidth.
The current lazy-loaded membership cache has three intertwined flaws:
1. It does not handle retries or clients re-using old tokens.
2. It incorrectly applies the membership cache to membership changes,
not just memberships added by lazy-loading.
3. It does not correctly handle multiple "sync streams" from the same
device.
In this PR we explicitly fix the first two issues in two commits, and
the third issue is mitigated:
1. Alongside each membership in the LL cache we store the `next_batch`
position of the response that carried it. A client that processed a
response syncs from its `next_batch` next. When a request arrives with a
`since` token, we delete every membership recorded after that token, as
they were sent in responses the client never processed. Memberships
recorded at or before `since` are kept.
2. We always return memberships that have changed, even if they appear
in the LL member cache.
These changes also half mitigate the issue of running multiple `/sync`
loops per-device (though we do not see this in the wild), as we'd at
least always return the changed membership states.
Fixes#19978 and #20278, supersedes #20279
Spawning from @Twi1ightSparkle
[spotting](https://matrix.to/#/!SGNQGPGUwtcPBUotTL:matrix.org/$vbqaJ6fJJBzMe1972Z_Ny8euVPWV8IxIS3YWGxP58r0?via=jki.re&via=element.io&via=matrix.org)
a fresh Synapse [running this query every 5
seconds](https://github.com/element-hq/synapse/blob/373fa7f542d86c1dbf82c4ae87c6bec8390e263a/synapse/storage/databases/main/client_ips.py#L440-L441)
which seemed excessive. My initial sniff test thought it was fine
because the more often we run the query, the smaller number of rows we
need to process at a time but @reivilibre brought up that we still have
to scan over all of the dead tuples each time which is a fixed cost
regardless.
Ideally, we'd instead fix the pagination of the query itself to avoid
re-scanning over the tuples. That's probably also a simple change and
I'm mostly just opening this PR to have a place to chuck my worked
example somewhere (mostly for myself). We can always have another
follow-up to do the proper fix.
### `user_ips` dead tuple calculations from `matrix.org`
For `matrix.org`, our autovacuum triggers after the table has
[~5%](https://github.com/matrix-org/matrix-ansible-private/blob/1b623c950e6b4db87f1dd17fa82c83be1b3b58cb/roles/postgres_role/templates/matrix-postgresql.conf.j2#L64-L66)
dead tuples (Postgres normally has a 20% default for
[`autovacuum_vacuum_scale_factor`](https://www.postgresql.org/docs/current/runtime-config-vacuum.html#GUC-AUTOVACUUM-VACUUM-SCALE-FACTOR))
The `user_ips` table on `matrix.org` has 21.8M rows so that means we
have to wait for `(21.8M * 0.05)` = ~1.1M dead rows to accumulate before
the autovacuum kicks in and cleans up all of the dead tuples.
Upper bound napkin math: If we assume that we've reached a steady state
where we prune just as many rows as we insert over the
`user_ips_max_age` time period (defaults to [28
days](https://github.com/element-hq/synapse/blob/373fa7f542d86c1dbf82c4ae87c6bec8390e263a/synapse/config/server.py#L697));
and if the vacuum only ran because of the prune: `21.8M / 28d ~= 778k` a
day -> takes ~1.4 days to accumulate enough dead tuples.
But most of the churn probably comes from updates to the `user_ips`
since every authenticated request updates `user_ips` [every 2
minutes](https://github.com/element-hq/synapse/blob/373fa7f542d86c1dbf82c4ae87c6bec8390e263a/synapse/storage/databases/main/client_ips.py#L52-L55).
And this matches reality:
For actual metrics of `user_ips` on `matrix.org` looking at the
[Postgres
metrics](https://grafana.matrix.org/d/000000009/postgres?orgId=1&var-data_source=000000001)
we have in Prometheus/Grafana:
- ~6.2M updates per day
(`increase(pg_stat_all_tables_n_tup_upd{schemaname!~"pg_.*",
schemaname!~"information_.*", instance=~"$instance"}[1d])`)
- ~792k inserts per day
(`increase(pg_stat_all_tables_n_tup_ins{schemaname!~"pg_.*",
schemaname!~"information_.*", instance=~"$instance"}[1d])`)
- (inserts don't create dead tuples)
- ~792k deletes per day
(`increase(pg_stat_all_tables_n_tup_del{schemaname!~"pg_.*",
schemaname!~"information_.*", instance=~"$instance"}[1d])`)
- 9.17 deletes/second
-> ~7M dead tuples per day
So we wait ~3.77 hours to trigger the next autovacuum for this table
`(1.1M * (7M / 24))`. Since the majority of the dead tuples are from
updates, those dead tuples on the other side of the `last_seen` index
and we probably don't have to scan over those for the prune loop. In
between vacuums, we still end up scanning ~124k dead tuples each time we
query on the prune though `(792k * (3.7/24))`.
So even for `matrix.org` levels of busyness and more aggressive
autovacuum, the `5s` interval overkill.
Given 9.17 deletes/second, if we choose a prune loop interval duration
of `60s`, it will pick-up ~550 rows which is under the query `LIMIT` set
(`5000`) with about an order of magnitude head-room to catch-up from
downtime or peak/heavy traffic.
Part of: MSC4354,
Part of: https://github.com/element-hq/synapse/issues/19641
This PR persists events in redacted form when they already have valid
redactions. That's not directly a sticky events-specific change (and I
feel it makes sense to redact immediately before persistence when we
already have the redaction).
This causes a nice effect which is that the event's stickiness is lost
immediately before persistence, because the `msc4354_sticky` field is
redacted by the redaction algorithm.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
To use `reactor.callFromThread` we need to take the GIL. However, the
vast majority of use-cases on the Rust side involve using that from
Tokio reactor threads, where we do not want to take the GIL (as it can
potentially block for a long time).
In #20252 we added a pure Rust alternative to `callFromThread`.
`run_python_awaitable` is the last usage of `callFromThread`, and so we
replace it with `dispatch_to_twisted`.
---
The diff is a lot smaller if you hide pure whitespace changes.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Our admin API to delete a room has a weird feature where you can supply
`new_room_user_id` and it will create a new room and join all of the
users to that room:
https://github.com/element-hq/synapse/blob/929e3524d189bf7d3402866b930cebde6bb54892/docs/admin_api/rooms.md#L640-L642
But that room can contain [suspended
users](https://spec.matrix.org/v1.19/client-server-api/#account-suspension)
which are unable to join new rooms. Currently, we use
`logger.exception(...)` for this failure which ends up in Sentry but
since this is probably expected outcome and we shouldn't elevate it to
that level of problem as it's just noise.
Spawning from seeing the error in Sentry,
https://sentry.tools.element.io/organizations/element/issues/11266279/?project=2
```
Traceback (most recent call last):
File "synapse/handlers/room.py", line 2532, in shutdown_room
await self.room_member_handler.update_membership(
File "synapse/handlers/room_member.py", line 675, in update_membership
result = await self.update_membership_locked(
File "synapse/handlers/room_member.py", line 794, in update_membership_locked
raise SynapseError(
SynapseError: 403: Joining rooms while account is suspended is not allowed.
```
Fixes#15871
### What was broken
Sending `{"auth": null}` to any endpoint that requires user-interactive
auth (for example `POST /keys/device_signing/upload`, `DELETE
/devices/{deviceId}` or `POST /register`) returned a 500 instead of the
usual 401 with the available flows. A non-object `auth` on endpoints
that do not validate the body with a model (such as `/register`) could
also 500, e.g. `{"auth": ["session"]}`.
### Root cause
`AuthHandler.check_ui_auth` did `clientdict.pop("auth", {})`, so the
`{}` default only applies when the key is missing. An explicit `null`
came through as `None` and the following `"session" in authdict` raised
`TypeError`. `get_session_id` had the same assumption that `auth` is a
dict.
### Pull Request Checklist
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Signed-off-by: Ankit Jha <ankit.jha@tradomate.one>
Spawning from
https://github.com/element-hq/synapse/pull/20143#discussion_r4080708454
`RestServlet.register` read `PATTERNS` via `getattr`, so it was typed as
`Any` and mypy never checked subclasses' patterns against what
`HttpServer.register_paths` expects.
This declares the attribute so that classes extending `RestServlet`
provide the right type for the `PATTERNS` attribute.
Add a metric for the number of times we see a stall due to being given a
future token.
This would have caught
https://github.com/element-hq/synapse/issues/20080, where lots of sync
streams got stuck due to a stream not getting replicated correctly.
## Summary
Clarify the `--exists-ok` command-line help text for
`register_new_matrix_user`.
The previous help text said:
> Do not fail if user already exists.
This could be interpreted as the existing user's account being updated.
The new wording explicitly states that the existing user account will
not be updated.
This also aligns the CLI help text with the existing Debian man page
documentation.
## Issue
Fixes#20112
## Testing
* `git diff --check`
* Reviewed the staged diff to confirm only the intended help-text change
was included.
Tokio tasks currently complete by taking the GIL and calling
`reactor.callFromThread`. Taking the GIL may block for a period of time,
and we do not want that to happen on the tokio reactor threads (as that
can block other work from happening).
To avoid this, we instead add a work queue that we can add to from Rust
without taking the GIL, which is drained by the reactor. We signal to
the Twisted reactor that it should wake up by using a unix socket pair,
which is exactly how `reactor.callFromThread` works.
---
The self-pipe trick is basically where you create a pair of unix sockets
connected to each other. One end is added to the reactor so the reactor
is woken up when there are bytes to read, and when another thread needs
to wake up the reactor it just needs to write a byte into the other unix
socket.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
This pull request fixes a few issues with the profile update stream,
when a user leaves a room. This is basically just mimicking what we
already had for when someone leaves a room, and they no longer share any
rooms, but in reverse - ie when we leave a room, and no longer share
rooms with some users. This was missed in the implementation when adding
the profile update stream for legacy and sliding sync.
[Fix missing profile update stream rows on user leaving
room](https://github.com/element-hq/synapse/commit/e953322226731a99718ebe13139b9c5f05ae4b62)
When a user leaves a room, profile update stream rows are generated with
the `LEFT_ROOM` action for each user in the room that no longer shares a
room with the user who left the room.
This also needs to happen in reverse. The user who left the room needs
to have a profile update stream row for each user they no longer share a
room with.
If we don't do this, clients may keep stale data around even after they
don't share a room with a user, which may mean they don't know when to
refetch profiles after re-joining a room with the stale profile data
user.
[Clear out old profile update stream rows when leaving a
room](https://github.com/element-hq/synapse/commit/e7642bf141b098c1eefaf74655e5e7e18415500b)
When we leave a room, ensure profile update stream rows are cleared out
for every user we no longer share a room with. This is the same as what
happens when someone else leaves a room, but in reverse. The only
remaining profile update stream row should be the `LEFT_ROOM` action.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
This is to prevent the case where URL previews return a MXC that
immediately 404s because the media in question has been quarantined.
This is mostly to help implementations which currently show an ugly
empty preview due to the MXC being sent down, despite being invalid.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Previously, the worker endpoint pattern included only the multi-item
lookup and `/restart`. Adjust the pattern so that it includes the
per-`delay_id` lookup endpoint as well.
Also adjust the worker documentation to use this same endpoint pattern.
Follow-up of #20210
[MSC4140](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/4140-delayed-events-futures.md),
"Account deactivation":
> when an account is deactivated, the homeserver MUST cancel that
account's delayed events which have not yet been added to a room's event
DAG. These cancelled records MAY be removed immediately, including their
stored event content, as an exception to the usual finalised-record
retention policy.
Deactivation currently leaves the user's scheduled delayed events,
including their content, in place, and they are still attempted at their
scheduled time:
| Delayed event scheduled by the user | What happens after deactivation
today |
| --- | --- |
| Message or state event, fires after deactivation has made the user
leave the room | The send fails with a 403 (user not in room) and the
record is dropped |
| Message or state event, fires before deactivation has made the user
leave the room (leaving happens room by room in the background) | The
event is sent |
| `m.room.member` join for themselves in a public room | The deactivated
user re-joins the room |
### What changes
- Deactivating an account (client or admin API) removes the user's
unsent delayed events as its first step, and re-arms the send timer for
whatever is scheduled next.
- Deactivation can run on a worker, while the send timer lives on the
main process, so the cancellation goes through a new replication
request.
- A delayed event whose send has already started is left alone, as with
a normal cancel: it is already on its way into the DAG, and the send
path removes its record itself.
Suspension and locking are unchanged.
### After #19038
Today a cancelled delayed event is simply deleted, so this PR deletes
the user's records too. #19038 changes cancellation to keep the record
and mark it as cancelled ("finalised"), so that clients can look up what
happened to a delayed event. Once it has landed, deactivation should
probably finalise the user's records as cancelled the same way, rather
than delete them. The MSC allows either: it permits removing the records
immediately, content included, as an exception to the usual retention of
finalised records, which is worth doing at least when the user asks to
be erased.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Ferrazzutti <af_0_af@hotmail.com>
Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
As I was working on https://github.com/element-hq/synapse/pull/20218, I
noticed what seemed to be an illegal configuration of synapse.
- `require_auth_for_profile_requests`: blocks profile requests unless
authenticated
- `limit_profile_requests_to_users_who_share_rooms`: blocks profile
requests unless authenticated user share a room with requested user
This, I think, should be an illegal config:
```
require_auth_for_profile_requests = false
limit_profile_requests_to_users_who_share_rooms = true
```
As of now, with such a config the shared-room check is never applied: a
profile can be requested anonymously, and also by an authenticated user
who doesn't share a room.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>
