The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
Fixes#19583
The nginx example in `docs/reverse_proxy.md` sets `proxy_set_header Host
$host:$server_port;`. The server block listens on 443, so nginx sends
`Host: matrix.example.com:443`. With `public_baseurl:
https://matrix.example.com/`, the request URI never starts with
`public_baseurl`, and `SsoRedirectServlet` keeps redirecting to itself.
That breaks SSO even when the internal and external ports are the same.
The line came in with #17994.
This switches the example to `proxy_set_header Host $host;`, as
@anoadragon453 recommended in #18359. I added a short comment for setups
where clients use a non-default port: use `$http_host` and put that port
in `public_baseurl`. The setup in #18359 (external port 50080) needs
that variant.
I left `docker/conf-workers/nginx.conf.j2` alone. Its listener port
matches `public_baseurl`, so `$host:$server_port` is right there.
This is a docs-only change, so there are no tests.
Thanks to @shibotto for reporting this and for pointing at the fix.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
It looks like our CI is sometimes being run with Python 3.15, where
`rpds-py` doesn't have wheels for our current version.
Unsure why it's intermittent.
Run:
https://github.com/element-hq/synapse/actions/runs/39464528041/job/118772376619
```
| error: failed to run custom build command for `pyo3-ffi v0.27.2`
|
| Caused by:
| process didn't exit successfully: `/tmp/tmp72_cy3_y/rpds_py-0.30.0/target/release/build/pyo3-ffi-ae84d9db989de8cd/build-script-build` (exit status: 1)
| --- stdout
| cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
| cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
| cargo:rustc-check-cfg=cfg(PyPy)
| cargo:rustc-check-cfg=cfg(GraalPy)
| cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
| cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
| cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
| cargo:rustc-check-cfg=cfg(Py_3_7)
| cargo:rustc-check-cfg=cfg(Py_3_8)
| cargo:rustc-check-cfg=cfg(Py_3_9)
| cargo:rustc-check-cfg=cfg(Py_3_10)
| cargo:rustc-check-cfg=cfg(Py_3_11)
| cargo:rustc-check-cfg=cfg(Py_3_12)
| cargo:rustc-check-cfg=cfg(Py_3_13)
| cargo:rustc-check-cfg=cfg(Py_3_14)
| cargo:rustc-check-cfg=cfg(Py_3_15)
| cargo:rerun-if-env-changed=PYO3_CROSS
| cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
| cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
| cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
|
| --- stderr
| error: the configured Python version (3.15) is newer than PyO3's maximum supported version (3.14)
| = help: this package is being built with PyO3 version 0.27.2
| = help: check https://crates.io/crates/pyo3 for the latest PyO3 version available
| = help: updating this package to the latest version of PyO3 may provide compatibility with this Python version
| = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
| warning: build failed, waiting for other jobs to finish...
| 💥 maturin failed
| Caused by: Failed to build a native library through cargo
| Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.15-64bit" PYO3_PYTHON="/tmp/tmpy90bal2g/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/tmpy90bal2g/.venv/bin/python" "cargo" "rustc" "--profile" "release" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmp72_cy3_y/rpds_py-0.30.0/Cargo.toml" "--lib"`
| Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmpy90bal2g/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with rpds-py (0.30.0)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "rpds-py (==0.30.0)".
PEP517 build of a dependency failed
```
```
| error: failed to run custom build command for `pyo3-ffi v0.26.0`
|
| Caused by:
| process didn't exit successfully: `/tmp/tmpy_lu888j/pydantic_core-2.41.5/target/release/build/pyo3-ffi-83252c0f9cb4430a/build-script-build` (exit status: 1)
| --- stdout
| cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
| cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
| cargo:rustc-check-cfg=cfg(PyPy)
| cargo:rustc-check-cfg=cfg(GraalPy)
| cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
| cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
| cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
| cargo:rustc-check-cfg=cfg(Py_3_7)
| cargo:rustc-check-cfg=cfg(Py_3_8)
| cargo:rustc-check-cfg=cfg(Py_3_9)
| cargo:rustc-check-cfg=cfg(Py_3_10)
| cargo:rustc-check-cfg=cfg(Py_3_11)
| cargo:rustc-check-cfg=cfg(Py_3_12)
| cargo:rustc-check-cfg=cfg(Py_3_13)
| cargo:rustc-check-cfg=cfg(Py_3_14)
| cargo:rustc-check-cfg=cfg(Py_3_15)
| cargo:rerun-if-env-changed=PYO3_CROSS
| cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
| cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
| cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
|
| --- stderr
| error: the configured Python interpreter version (3.15) is newer than PyO3's maximum supported version (3.14)
| = help: please check if an updated version of PyO3 is available. Current version: 0.26.0
| = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
| warning: build failed, waiting for other jobs to finish...
| 💥 maturin failed
| Caused by: Failed to build a native library through cargo
| Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.15-64bit" PYO3_PYTHON="/tmp/tmprci5om5_/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/tmprci5om5_/.venv/bin/python" "cargo" "rustc" "--profile" "release" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmpy_lu888j/pydantic_core-2.41.5/Cargo.toml" "--lib" "--crate-type" "cdylib"`
| Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmprci5om5_/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with pydantic-core (2.41.5)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "pydantic-core (==2.41.5)".
```
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Lays the two foundations the Rust DBAPI2 backend needs before any
connection code: a Python↔Postgres value-mapping layer, and the helpers
that drive async work from sync Python.
Value mapping:
- `PgValue` — enum that implements `tokio_postgres::types::ToSql`,
encoding into Postgres' binary wire format per the prepared-statement
column type (so a single Python `int` becomes INT2/INT4/INT8 with range
checks).
- `PythonPgFromSql` — the decode counterpart, turning wire bytes back
into Python objects (SQL NULL → `None`).
- Covers int / float / bool / str / bytes / None; lists etc are left to
a follow-up.
Blocking helpers:
- `block_on` / `block_on_pg_result` / `block_on_next` drive
`tokio-postgres` futures to completion from sync, GIL-holding Python
methods, releasing the GIL for the wait.
Both modules are `pub` for now so not-yet-consumed items don't trip
clippy's `dead_code` lint; later PRs in the stack tighten visibility.
Everything here is unit-tested with no live Postgres server.
## What the MSC says
From MSC4140's [Auth rule
evaluation](https://github.com/matrix-org/matrix-spec-proposals/blob/b7cb09664fed684388ba656459d08cfaf786c1d1/proposals/4140-delayed-events-futures.md#auth-rule-evaluation):
> The server SHOULD also check the event could be sent when it is
scheduled and reject the request if not. In particular, servers SHOULD
check the sender is in the room (or otherwise authorized to send the
event) to avoid storing events for rooms the user is not in.
## This PR
- Checks the event against the room's auth rules as if it were sent now,
and returns the same 403 `M_FORBIDDEN` that `/send` and `/state` would.
- Changes to the sender's own membership are exempt: a join or knock can
be sent from outside the room, which is how this PR reads the MSC's "or
otherwise authorized". They are checked by the auth rules when they are
sent.
Here are a few examples:
| Delayed event | Before | After |
| ----------------------------------------------------------------- |
--------------------------------------- |
---------------------------------------- |
| `m.room.message` from outside the room | 200 `delay_id`, fails at send
time | 403 `M_FORBIDDEN` |
| Kick of another member from outside the room | 200 `delay_id`, fails
at send time | 403 `M_FORBIDDEN` |
| `m.room.name` from a member whose power level is too low | 200
`delay_id`, fails at send time | 403 `M_FORBIDDEN` |
| **Exception**: join of a public room from outside it | 200 `delay_id`,
joins at send time | unchanged |
| Join from outside the room, then a message due after it | both
accepted, message sent once joined | join accepted, message 403
`M_FORBIDDEN` |
Part of: MSC4354 (Sticky Events, Experimental Feature tracking issue
#19409)
When a user is the sender of a sticky event that comes down sync, that
user is now in scope for having their lazy-loaded membership sent down
in sync.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Part of: MSC4354 (Experimental Feature tracking issue #19409)
Follows: #20165 (Sticky Event federation backlog + catch-up
transactions)
The MSC says that we need to tell a newly-joining server about all of
our sticky events (yes, even the ones before the server joined the
room).
This PR introduces the mechanism, building on the 'Sticky Event backlog'
mechanism introduced in #20165, to make that happen.
Most of the logic is taken from the original Sticky Events PR #18968,
but the Sticky Event backlog system did not exist back then.
(The benefit of having the persistent Sticky Event backlog is that we
don't need to rely on the federation senders not crashing to make sure
we get around to sending the Sticky Events out.)
- Add NEW_SERVER_JOINED replication command. (This is a cross-worker
'wake-up call' to check for a sticky event backlog)
We send it when persisting a state change that causes a new server to be
in the room.
We handle it by waking up federation senders for the destination,
including checking the sticky event backlog tables.
- When a server newly joins a room, we insert a sticky event backlog row
for that server, if there are any Sticky Events needing to be sent.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Depends on: https://github.com/matrix-org/complement/pull/914
`displayname` and `avatar_url` already did not permit nulls, seemingly
from #8223.
The specification at v1.15 does not permit `null` for those fields
either, so this seems consistent with historical specification.
This only changes behaviour for 'custom' fields.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Follows: #18375
This bugfix is a prerequisite for MSC4354 (Sticky Events, Experimental
Feature tracking issue #19409) changes.
- Add a failing test for join->join being newly_joined
- Fix the join->join transition being treated as a new join
(when prev_events didn't coincidentally mask the problem)
Rationale:
`event_edges` / `prev_events` does NOT link a membership event
to its previous state.
(It could even be someone else's membership!)
In most real-world cases, this would generate a `prev_membership` of
NULL
for every normal membership change.
It appears as though the author intended some kind of 'previous state'
relationship, but if you fix the first branch of the `UNION` query
to use `current_state_delta_stream` for this (which is the source of
state transitions), then it looks very redundant with the second
branch.
It looks like the real reason we want to use the snapshots table is for
out-of-band memberships (outlier memberships).
So we continue using that for those, but ONLY for those.
The `SUSPICIOUS` if block is removed because it appears to have been
papering over this issue by deduplicating the spurious `NULL` ->
non-null transitions,
but when you remove those duplicates at the source, it causes more
trouble
because it doesn't know that what it's 'deduplicating' is actually the
same membership transition.
This means if you join, then out-of-band leave (e.g. last member on the
server
to leave), that room would appear in `newly_joined` on a new initial
sync.
(Tests did in fact catch this, too.)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Fix https://github.com/element-hq/synapse/issues/20347
See
https://github.com/element-hq/synapse/issues/20347#issuecomment-6154861014
for more details.
Caused by the latest release of `txredisapi@1.6.0` (2026-10-09) which included https://github.com/IlyaSkriblovsky/txredisapi/pull/162
> May break compatibility with some specific user code because of inheritance changes.
And in the changelog:
> ### Incompatible changes
>
> - Connections are maintained by
> `twisted.application.internet.ClientService` now, and `RedisFactory` is no
> longer a `ReconnectingClientFactory`. Code connecting a factory on its own
> with `reactor.connectTCP()` or `twisted.application.internet.TCPClient` has
> to use `factory.startConnecting(endpoint)` instead - see `examples/subscriber.py`
>
> - `factory.continueTrying = False` is replaced by `factory.stopTrying()`;
> `retry()`, `resetDelay()` and `delay` are gone. `maxDelay`, `initialDelay`,
> `factor` and `jitter` still configure the backoff and are used by the new
> `RedisFactory.retryDelay()`
>
> - `SentinelConnectionFactory.try_to_connect()` is gone: the address of the
> master or of a slave is discovered by the endpoint before every attempt.
> Subclasses overriding `try_to_connect()`, `clientConnectionFailed()` or
> `clientConnectionLost()` have no effect anymore
>
> [...]
>
> *-- [`IlyaSkriblovsky/txredisapi` -> *Release 1.6.0 (2026-10-09)*](https://github.com/IlyaSkriblovsky/txredisapi/blob/f844759b8aabc689eb87bf300ed3c675b84c3608/CHANGELOG.md#L31-L35)*
### Testing strategy
1. Check out the PR: `git checkout
madlittlemods/20347-txredisapi-constraint`
1. Install into a fresh scratch venv:
```shell
rm -rf /tmp/venv && python3 -m venv /tmp/venv
/tmp/venv/bin/pip install '.[all]'
/tmp/venv/bin/pip show txredisapi | grep Version # expect 1.5.0 with
this PR
```
1. One-time setup: make a homeserver config with Redis enabled:
```shell
mkdir -p /tmp/hs && cd /tmp/hs
/tmp/venv/bin/python -m synapse.app.homeserver --server-name repro.test
\
-c homeserver.yaml --generate-config --report-stats=no
printf '\nredis:\n enabled: true\n host: localhost\n' >> homeserver.yaml
cd -
```
- On my machine, I also have to remove the IPv6 bind for `::1`, so `sed
-i '/^ - ::1$/d' /tmp/hs/homeserver.yaml`
1. Run Synapse: `/tmp/venv/bin/python -P -m synapse.app.homeserver -c
/tmp/hs/homeserver.yaml`
- It should run successfully with this PR. You can also check
`/tmp/hs/homeserver.log` for logs.
Repeat these steps on `develop` to see the original failure described in
https://github.com/element-hq/synapse/issues/20347 (`AttributeError:
'super' object has no attribute 'startedConnecting'. Did you mean:
'startConnecting'?`)
Gets one step further in the job started by
https://github.com/matrix-org/synapse/pull/12893
Part of: https://github.com/element-hq/synapse/issues/20026
## Manual testing strategy
Unfortunately it's quite difficult to actually trigger this.
You need to:
- check out an old full_schema, e.g. 54 which was present when `v1.62.0`
was released
- remove the version 72 `full_schema`, as it already has the background
update applied
- remove some deltas that are broken when upgrading across such a large
version gap (!) https://github.com/element-hq/synapse/issues/20070
- then run any trial test with Postgres enabled to prove that the
database schema applies
```
git checkout v1.62.0 -- synapse/storage/schema/state/full_schemas/54
git checkout v1.62.0 -- synapse/storage/schema/main/full_schemas/54
git checkout v1.62.0 -- synapse/storage/schema/common/full_schemas/54
rm -r synapse/storage/schema/*/full_schemas/72
rm synapse/storage/schema/main/delta/58/10_pushrules_enabled_delete_obsolete.sql
rm synapse/storage/schema/main/delta/79/04_mitigate_stream_ordering_update_race.py
rm synapse/storage/schema/main/delta/56/redaction_censor3_fix_update.sql.postgres
rm synapse/storage/schema/main/delta/60/01recreate_stream_ordering.sql.postgres
SYNAPSE_TEST_LOG_LEVEL=DEBUG SYNAPSE_POSTGRES=1 poetry run trial tests.handlers.test_sliding_sync.ComputeInterestedRoomsTestCase_new.test_no_joins_after_to_token
```
As light validation, I purposefully inserted a SQL error into the query
to check that it is caught by the above testing strategy.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Closes#20116
> [!NOTE]
> This PR Follows #20115 which already implemented this for
`m.device_list_update`
> User IDs with localparts containing characters outside the range
U+0021 to U+007E, or with
> an empty localpart, are considered non-compliant. For current room
versions, servers must
> still accept events using such user IDs over federation; however they
SHOULD NOT forward
> such user IDs to clients when referenced outside the context of an
event. For example,
> device list updates from non-compliant user IDs would be dropped by
the receiving server.
>
> — [Spec
1.19](https://spec.matrix.org/v1.19/appendices/#historical-user-ids)
This PR prevents from forwarding non compliant user ids for the
following events:
- `m.signing_key_update` 8c3c35e95d
- `m.presence` 26a8dfe862
- `m.typing` 1110cf2d73
- `m.receipt` 9e2da8b0a6
- `m.direct_to_device` 9541232a30
These things are not needed now that MSC2965 has stabilised. In any
case, they
should never have been exposed without a config flag in the first place.
Eric Eastwood