Check that `/proc/<tid>/stat` parses correctly (including a thread name
with spaces and brackets), that main-thread and registered-thread CPU
land under the right `thread_type`, and that the per-thread totals add
up to the process's CPU time between two scrapes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SXqkiTqEA9Nb2ycHaArDBb
Modules such as the S3 storage provider run work on their own Twisted
threadpool through `ModuleApi.defer_to_threadpool`. Register the pool
with the thread CPU collector there, under its Twisted name, so the S3
provider's threads show up as `threadpool_s3-pool` rather than `other`.
A threadpool now keeps the name it was first registered under, so a
module passing the reactor's pool does not rename `threadpool_default`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDruzqbGMzEHotouLuudxP
jaeger-client sends spans from a thread it starts itself, which was
reported as `other`. Add `register_thread` to the thread CPU collector
and register the threads that `initialize_tracer` starts as
`jaeger-reporter`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDruzqbGMzEHotouLuudxP
`synapse_thread_cpu_seconds_total{thread_type, mode}` reads
`/proc/self/task` on each scrape and groups threads into the main thread,
each registered threadpool, tokio (by kernel thread name), and `other`.
The kernel only reports CPU time for live threads, so the collector sums
per-thread deltas between scrapes. A thread that exits loses the CPU it
used since the last scrape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDruzqbGMzEHotouLuudxP
The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
The problem is that while `pillow@12.3.0` supports Python >= 3.10,
`pillow@13.0.0` only supports Python > 3.11, and we haven't dropped
support for Python 3.10 yet (tracked by
https://github.com/element-hq/synapse/issues/20041).
`pillow@13.0.0` includes some important security fixes:
https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html
### How was this change made?
1. `poetry update pillow`
1. Manually change the `"pillow"` `markers` in `poetry.lock` to split at
Python 3.11 instead of 3.14
Please note this is not a durable change. `poetry` will try to revert
this any time we try to make a change (requires diligence to make sure
we keep changing it back).
This maintenance burden will disappear once we drop Python 3.10 support
(tracked by https://github.com/element-hq/synapse/issues/20041).
### Alternatives
If we were up for updating the minimum version of `pillow` in
`pyproject.toml`, we could do something like the following which would
resolve in `poetry.lock` durably how we want,
`pyproject.toml`
```
# Pillow 13 dropped Python 3.10 but has security fixes. Split so 3.11+ can pick up
# the security fixes.
#
# FIXME: Remove the split once we drop Python 3.10 support.
"Pillow>=10.0.1,<13; python_version < '3.11'",
"Pillow>=13.0.0; python_version >= '3.11'",
```
It looks like we [previously
bumped](https://github.com/element-hq/synapse/blob/4853cd4238e100183fd7b0852ddf3e8eef894658/pyproject.toml#L54-L56)
the minimum version of `Pillow` for another CVE
(https://github.com/matrix-org/synapse/pull/16347)
Although, I'm guessing we'd rather leave the minimum version alone and
only update `poetry.lock`.
I think the problem with updating the minimum is that the packagers for
Linux distros will have to deal with not having the latest `pillow`
version available and may patch `pillow` a different way.
Fixes#19583
The nginx example in `docs/reverse_proxy.md` sets `proxy_set_header Host
$host:$server_port;`. The server block listens on 443, so nginx sends
`Host: matrix.example.com:443`. With `public_baseurl:
https://matrix.example.com/`, the request URI never starts with
`public_baseurl`, and `SsoRedirectServlet` keeps redirecting to itself.
That breaks SSO even when the internal and external ports are the same.
The line came in with #17994.
This switches the example to `proxy_set_header Host $host;`, as
@anoadragon453 recommended in #18359. I added a short comment for setups
where clients use a non-default port: use `$http_host` and put that port
in `public_baseurl`. The setup in #18359 (external port 50080) needs
that variant.
I left `docker/conf-workers/nginx.conf.j2` alone. Its listener port
matches `public_baseurl`, so `$host:$server_port` is right there.
This is a docs-only change, so there are no tests.
Thanks to @shibotto for reporting this and for pointing at the fix.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
It looks like our CI is sometimes being run with Python 3.15, where
`rpds-py` doesn't have wheels for our current version.
Unsure why it's intermittent.
Run:
https://github.com/element-hq/synapse/actions/runs/39464528041/job/118772376619
```
| error: failed to run custom build command for `pyo3-ffi v0.27.2`
|
| Caused by:
| process didn't exit successfully: `/tmp/tmp72_cy3_y/rpds_py-0.30.0/target/release/build/pyo3-ffi-ae84d9db989de8cd/build-script-build` (exit status: 1)
| --- stdout
| cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
| cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
| cargo:rustc-check-cfg=cfg(PyPy)
| cargo:rustc-check-cfg=cfg(GraalPy)
| cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
| cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
| cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
| cargo:rustc-check-cfg=cfg(Py_3_7)
| cargo:rustc-check-cfg=cfg(Py_3_8)
| cargo:rustc-check-cfg=cfg(Py_3_9)
| cargo:rustc-check-cfg=cfg(Py_3_10)
| cargo:rustc-check-cfg=cfg(Py_3_11)
| cargo:rustc-check-cfg=cfg(Py_3_12)
| cargo:rustc-check-cfg=cfg(Py_3_13)
| cargo:rustc-check-cfg=cfg(Py_3_14)
| cargo:rustc-check-cfg=cfg(Py_3_15)
| cargo:rerun-if-env-changed=PYO3_CROSS
| cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
| cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
| cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
|
| --- stderr
| error: the configured Python version (3.15) is newer than PyO3's maximum supported version (3.14)
| = help: this package is being built with PyO3 version 0.27.2
| = help: check https://crates.io/crates/pyo3 for the latest PyO3 version available
| = help: updating this package to the latest version of PyO3 may provide compatibility with this Python version
| = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
| warning: build failed, waiting for other jobs to finish...
| 💥 maturin failed
| Caused by: Failed to build a native library through cargo
| Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.15-64bit" PYO3_PYTHON="/tmp/tmpy90bal2g/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/tmpy90bal2g/.venv/bin/python" "cargo" "rustc" "--profile" "release" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmp72_cy3_y/rpds_py-0.30.0/Cargo.toml" "--lib"`
| Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmpy90bal2g/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with rpds-py (0.30.0)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "rpds-py (==0.30.0)".
PEP517 build of a dependency failed
```
```
| error: failed to run custom build command for `pyo3-ffi v0.26.0`
|
| Caused by:
| process didn't exit successfully: `/tmp/tmpy_lu888j/pydantic_core-2.41.5/target/release/build/pyo3-ffi-83252c0f9cb4430a/build-script-build` (exit status: 1)
| --- stdout
| cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
| cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
| cargo:rustc-check-cfg=cfg(PyPy)
| cargo:rustc-check-cfg=cfg(GraalPy)
| cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
| cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
| cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
| cargo:rustc-check-cfg=cfg(Py_3_7)
| cargo:rustc-check-cfg=cfg(Py_3_8)
| cargo:rustc-check-cfg=cfg(Py_3_9)
| cargo:rustc-check-cfg=cfg(Py_3_10)
| cargo:rustc-check-cfg=cfg(Py_3_11)
| cargo:rustc-check-cfg=cfg(Py_3_12)
| cargo:rustc-check-cfg=cfg(Py_3_13)
| cargo:rustc-check-cfg=cfg(Py_3_14)
| cargo:rustc-check-cfg=cfg(Py_3_15)
| cargo:rerun-if-env-changed=PYO3_CROSS
| cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
| cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
| cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
|
| --- stderr
| error: the configured Python interpreter version (3.15) is newer than PyO3's maximum supported version (3.14)
| = help: please check if an updated version of PyO3 is available. Current version: 0.26.0
| = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
| warning: build failed, waiting for other jobs to finish...
| 💥 maturin failed
| Caused by: Failed to build a native library through cargo
| Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.15-64bit" PYO3_PYTHON="/tmp/tmprci5om5_/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/tmprci5om5_/.venv/bin/python" "cargo" "rustc" "--profile" "release" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmpy_lu888j/pydantic_core-2.41.5/Cargo.toml" "--lib" "--crate-type" "cdylib"`
| Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmprci5om5_/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with pydantic-core (2.41.5)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "pydantic-core (==2.41.5)".
```
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Erik Johnston