An out-of-band leave, such as a remote invite rescinded by the inviter or
a rejected remote invite, is an outlier and so not in the room's
timeline. Incremental sync already sends it as the room's timeline, but
initial sync only loaded the timeline from the room stream, so it
dropped the room if the user had never been in it, and otherwise
returned the history without the leave.
Initial sync now loads the room's history before the out-of-band leave
and adds the leave at the end, including when the history has a gap.
When the timeline starts with the leave, the state at the end of the
timeline is used as the state at its start, as the leave has no state of
its own.
Rescinded invites are returned whether or not `include_leave` is set,
like any other room the user was kicked from, as incremental sync
already did.
This made Complement's `Inviter_user_can_rescind_invite_over_federation`
flaky: if the rescission reached the invitee's server before the test's
first sync, the leave was never returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fixes#19257.
When an identity server returns a non-2xx response to
`/_matrix/identity/v2/store-invite` during a third-party invite,
`ask_id_server_for_third_party_invite` lets the `HttpResponseException`
escape. The client receives a 500 `M_UNKNOWN` error and the real cause
(for example, `M_THREEPID_IN_USE`) is lost.
This change catches `HttpResponseException` and converts it with
`to_synapse_error()`, which matches how the other identity server calls
in `IdentityHandler` (bind, `requestToken`) already handle error
responses.
A new test in `ThreepidInviteTestCase` makes the identity server return
a 400 `M_THREEPID_IN_USE` error and checks that the client receives the
same code and errcode. The test fails on `develop` with `500 != 400` and
passes with this change.
Broke by #20289. Hopefully fixeselement-hq/element-web#35567.
One subtle change in #20289 was that we started to filter rooms with
`full_state` set through the cache. This *should* be safe since none of
the membership events should have been previously sent down the sync.
However, there is a bug with the lazy-loaded member cache when two
concurrent syncs for the same device happens. The updates to the cache
from the handling of one request are immediately seen by the other
request, polluting the cache.
This is a relatively common scenario in Element Web when a user reloads
the page. The sync from before the reload is still being processed when
Synapse receives the initial sync from after the reload.
The full fix to correctly handle overlapping requests is a separate PR,
but this is a stop-gap measure that should help Element Web.
Small, easy, low-hanging suggestions when wondering how to speed up our
release process.
- CI: Use multithreading in tarball xz compression
It looks like this compression takes about ~2 minutes. If we use 4 cores
then maybe this will be ~30 seconds + nonlinearity overhead.
- CI: Use trial --jobs=6 instead of -j2 for debs
Other than matching regular CI, this might take off 3~4 min.
Other than that, some caching for the release artifacts step and using
bigger runners to speed up the tests more might help, but those are more
invasive changes.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Checking whether a username is available looks users up
case-insensitively, with `WHERE lower(name) = lower(?)`. But there is
currently no index on `lower(name)`, so each check scans the whole
table.
This backs registration, `GET /_synapse/admin/v1/username_available` and
`GET /_synapse/mas/is_localpart_available`, which Matrix Authentication
Service will call as the user types in its registration form
([Interactive feedback for username availability in the registration
form
(element-hq/matrix-authentication-service#5977)](https://github.com/element-hq/matrix-authentication-service/pull/5977)).
- Adds `users_lower_name` on `users (lower(name))`, built as a
background update, on both PostgreSQL and SQLite.
- matrix.org already has an index with this name and definition, created
by hand ([Postgres indexes differ on matrix.org
(matrix-org/synapse#11893)](https://github.com/matrix-org/synapse/issues/11893)).
The background update drops and rebuilds it concurrently rather than
adding a duplicate; lookups fall back to a table scan until the rebuild
finishes.
### Pull Request Checklist
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
This includes the following:
- Implements ['Implicit Forward
Pagination'](https://github.com/matrix-org/matrix-spec/issues/2378).
Remove the extension's `since` and `next_batch` fields. The position is
now tracked via the global `pos`, with per-room connection state
recording how far each room has been sent so that `limit` paginates
without holding `pos` back and lost responses are re-delivered.
- Discussed in
https://github.com/matrix-org/matrix-spec-proposals/pull/4480#discussion_r4165887827
- Honour the `lists` and `rooms` fields.
- Send all unexpired sticky events when a room enters scope for the
first time or is newly joined, and catch up on events sent while a room
was out of scope when it re-enters.
- Enabling the extension partway through a connection returns all
unexpired sticky events for in-scope rooms, and re-enabling it after
disabling it returns the unexpired sticky events sent in the meantime.
I tried to split this up into standalone commits though the bulk of the
change is in the third commit.
---------
Co-authored-by: Olivier 'reivilibre <git.oo@emunest.net>
The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
Fixes#19583
The nginx example in `docs/reverse_proxy.md` sets `proxy_set_header Host
$host:$server_port;`. The server block listens on 443, so nginx sends
`Host: matrix.example.com:443`. With `public_baseurl:
https://matrix.example.com/`, the request URI never starts with
`public_baseurl`, and `SsoRedirectServlet` keeps redirecting to itself.
That breaks SSO even when the internal and external ports are the same.
The line came in with #17994.
This switches the example to `proxy_set_header Host $host;`, as
@anoadragon453 recommended in #18359. I added a short comment for setups
where clients use a non-default port: use `$http_host` and put that port
in `public_baseurl`. The setup in #18359 (external port 50080) needs
that variant.
I left `docker/conf-workers/nginx.conf.j2` alone. Its listener port
matches `public_baseurl`, so `$host:$server_port` is right there.
This is a docs-only change, so there are no tests.
Thanks to @shibotto for reporting this and for pointing at the fix.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Devon Hudson