Give tokio tasks a captured logcontext, resolving the module-doc TODO:
- LogContextHandle is a cheap, clone-able, GIL-free handle in the same
Option<Py<LoggingContext>> representation the storage slots use.
create_deferred captures the caller's context at the FFI boundary and
scopes it onto the spawned task via a tokio task-local, which rides with
the task across .await points. current_context() gives the task-local
read precedence, so log records emitted while a task is polled — via
LoggingContextFilter and pyo3-log — are attributed to the captured
context with no per-record stamping.
- The switch primitive is only ever driven on reactor/threadpool threads,
never during a tokio-scoped poll (where the write would be invisible to
reads); swap_current_context enforces that invariant with an error log
rather than trusting it.
- run_python_awaitable restores the captured context on the reactor
thread before driving the awaitable, so Python called back from Rust
(e.g. DatabasePool.runInteraction from the Rust /versions handler) runs
in — and accounts its DB usage against — the right request. The restore
protocol lives in a new with_logcontext helper (the Rust equivalent of
`with PreserveLoggingContext(...)`): an error cannot skip the restore
(which would leak the context onto the reactor thread permanently), and
a context that has already finished is not re-started (create_deferred
does not propagate cancellation, so a task can outlive its request; see
the TODO) — such work runs in the sentinel instead.
- tests/synapse_rust/test_logcontext.py exercises both guarantees through
real production code paths: reqwest's log records carry the caller's
request id, and the /versions handler's DB transaction lands on the
caller's usage accounting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFbRtswu7rsHrttJFauUUb
The "current logcontext" slot moves from a Python threading.local into
the Rust extension, together with a native port of LoggingContext. A
Python thread-local is invisible to Rust: each tokio worker thread would
see its own slot, permanently at the sentinel, so logging emitted from
Rust could not be attributed to the request that caused it. This lays
the storage groundwork; a follow-up change gives tokio tasks a
task-scoped capture (see the module-doc TODO).
Design notes, for review:
- The slot is typed: Option<Py<LoggingContext>>, with None representing
the sentinel. The _Sentinel class and SENTINEL_CONTEXT singleton stay
pure Python, unchanged; thin wrappers on current_context and
set_current_context in synapse.logging.context convert between the
singleton and None at the boundary, so no Rust code ever sees or
produces the sentinel object. pyo3's extraction enforces the type:
anything that is not a LoggingContext (or subclass) or None raises
TypeError.
- The accounting policy is native too: set_current_context reads the
thread rusage once via libc (no per-switch struct_rusage allocation) and
runs the stop/start bookkeeping inline for base LoggingContexts, only
dispatching through Python for subclasses (BackgroundProcessLoggingContext)
so their overrides run. start()/stop() now take an
Optional[tuple[float, float]] instead of a struct_rusage, the
get_thread_resource_usage/is_thread_resource_usage_supported/
get_thread_id module helpers are gone, LoggingContext.previous_context
is now Optional[LoggingContext] (None where it used to hold
SENTINEL_CONTEXT), and the nominally-private _resource_usage attribute
is no longer exposed (nothing read it; use get_resource_usage()) —
worth an upgrade note when this is released, as out-of-tree code may
rely on the old shapes.
- The switch path avoids per-operation allocation and Python round-trips:
names are stored as Py<PyString> (LoggingContextFilter reads
server_name and str(context) per log record process-wide, now
INCREF-only), error messages materialise the context name only in the
cold branches, and the thread id is read via PyThread_get_thread_ident
(the exact value threading.get_ident() returns) rather than by calling
into Python.
- The attribute surface, method set and error-message wording are a
compatibility contract, pinned by the characterization tests (which now
exercise the tuple-based start/stop API).
- The opt-in synapse.logging.context.debug switch traces are emitted from
Rust via pyo3-log, whose level cache only refreshes on
reset_logging_config(); docs/log_contexts.md documents the manhole
procedure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFbRtswu7rsHrttJFauUUb
This does two things, first it adds a config flag to ignore rooms for
the purposes of presence routing.
Secondly, it changes the caching behaviour to try and improve the cache
hit ratio. Previously, the size of the `do_users_share_a_room` cache
(which stores pairs of users) needs to `O(n²)` for the number of online
users, which is infeasible for large servers.
Instead, we call `get_users_in_room` for both the syncing and updated
users. This sounds more expensive, but a) we will already have cached
the syncing user's rooms, and b) we will only calculate the updated
user's rooms once (rather than once per syncing user).
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds `last_active_granularity`, `sync_online_timeout` and `idle_timeout`
options to the `presence` config section, controlling the previously
hard-coded `LAST_ACTIVE_GRANULARITY`, `SYNC_ONLINE_TIMEOUT` and
`IDLE_TIMER` constants (which remain as the defaults).
This is mainly useful on deployments that ratelimit how often syncs can
affect presence (`rc_presence`): the sync timeout must exceed the
ratelimit interval or users flap offline between syncs, so tuning down
presence traffic currently requires squeezing under the fixed 30s limit.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
After looking into it, just a couple of things to pick a bone at in the
old wording,
which I thought could be clarified for when I next come to look at this
again.
- the claim that there's a fundamental difference; I'd argue there isn't
really, it's just by convention
on some mainstream distros. So I have changed this to 'typically'
- statements that some distros fetch dependencies at build time
(probably does happen, but
traditional distros make a point of not doing this for the reasons you'd
expect).
- This was probably meant to be talking about Debian, but my observation
based on sample size of 3 is that some crates are packaged natively,
others are vendored in the respective application's source package (like
they do for us) and sometime they patch the bounds a bit
There could probably be room to talk about how distros vendoring
packages is a maintenance burden on them,
but I guess it's a bit moot as we would struggle to conform to wide
enough bounds to make everyone
happy (and anyway; I expect the distros that vendor packages have the
tooling to make this easy to
update and we do keep on top of security updates and release
frequently...)
---
Spawning from discussion in
[`#element-backend-internal:matrix.org`](https://matrix.to/#/!SGNQGPGUwtcPBUotTL:matrix.org/$VttYPPUevn2S_W_rrzg2ZOXWI6aKebk2ganTgrLEWUc?via=jki.re&via=element.io&via=matrix.org)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
The reason for querying this support was wanting support for SQLite's JSON operators, which are currently not present in the SQLite version found in Ubuntu's oldest supported LTS.
The JSON operators were used in some of the sticky events work (related: #19452).
Our ruling was that we should support Ubuntu oldest LTS equally to Debian oldstable, so support the oldest of the two versions from those.
That makes some kind of sense as it would be difficult to do otherwise without dropping support for that version of Ubuntu altogether, given if we kept publishing packages intended for use with Postgres, there's a risk that an innocent sysadmin would update their SQLite deployment without realising that it is no longer supported.
This was [discussed months ago
(private)](https://docs.google.com/document/d/12RZKPk3a4__JUSH9wYHODo9rRyKzsHg6BSCAcmqmbOU/edit?tab=t.0#bookmark=id.fcdvoc88dy5s)
and at [private](https://docs.google.com/document/d/12RZKPk3a4__JUSH9wYHODo9rRyKzsHg6BSCAcmqmbOU/edit?tab=t.0#bookmark=id.u48ivjge4qpt).
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Knowledge from @reivilibre in
[`#element-backend-internal:matrix.org`](https://matrix.to/#/!SGNQGPGUwtcPBUotTL:matrix.org/$2WUfCdA02wZw-6-jhw3QbLQ44BmKJrqLuZ6wjz2r7hk?via=jki.re&via=element.io&via=matrix.org)
on 2025-12-15.
Spawning from me making Rust changes but nothing useful was printed
until I added `-v`,
```shell
$ poetry install --extras all
Installing dependencies from lock file
Package operations: 0 installs, 1 update, 0 removals
- Updating pyjwt (2.11.0 -> 2.12.0)
Installing the current project: matrix-synapse (1.154.0rc1)
Failed to install /home/eric/Documents/github/element/synapse
```
I also see `poetry run maturin develop` suggested but I'd prefer not to
need to install `maturin` as yet another system tool to manage myself.
Added details how synapse syncs the picture claim when
update_profile_information setting is true. Addresses #17836
---------
Co-authored-by: Michael Hlas <3398654+mhlas7@users.noreply.github.com>
Fixes: #19616
This caused 2+ people trouble now, so worth batting away with a
low-effort change if we can.
Only seen on macOS so far, but nothing stops SQLite being configured in
defensive mode by default on other platforms, so it is not necessarily
entirely specific to macOS.
We *could* also do this for Python < 3.12 but it'd be more effort and I
don't know if it's worth it.
(For context @kegsay says the interpreter with this problem was
installed through `pyenv install`.)
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This moves the dev dependencies to PEP 735 dependency groups, to help us
move to standard project metadata, which will help us moving to `uv`
(#19566)
This requires poetry 2.2.0
It's pretty hard to remember the order of all of these ambiguous
numbers. I assume they're not totally labeled already to cut down on the
length when scanning with your eyes. This just adds a few hints of what
each grouping is.
Spawning from [staring at some Synapse
logs](https://github.com/element-hq/matrix-hosted/issues/10631) and
cross-referencing the Synapse source code over and over.
Fixes: #19540Fixes: #16290 (side effect of the proposed fix)
Closes: #12804 (side effect of the proposed fix)
Introduced in: https://github.com/matrix-org/synapse/pull/8932
---
This PR is a relatively simple simplification of the profile change on
deactivation that appears to remove multiple bugs.
This PR's **primary motivating fix** is #19540: when a user is
deactivated and erased, they would be kept in the user directory. This
bug appears to have been here since #8932 (previously
https://github.com/matrix-org/synapse/pull/8932) (v1.26.0).
The root cause of this bug is that after removing the user from the user
directory, we would immediately update their displayname and avatar to
empty strings (one at a time), which re-inserts
the user into the user directory.
With this PR, we now delete the entire `profiles` row upon user erasure,
which is cleaner (from a 'your database goes back to zero after
deactivating and erasing a user' point of view) and
only needs one database operation (instead of doing displayname then
avatar).
With this PR, we also no longer send the 2 (deferred) `m.room.member`
`join` events to every room to propagate the displayname and avatar_url
changes.
This is good for two reasons:
- the user is about to get parted from those rooms anyway, so this
reduces the number of state events sent per room from 3 to 1. (More
efficient for us in the moment and leaves less litter in the room DAG.)
- it is possible for the displayname/avatar update to be sent **after**
the user parting, which seems as though it could trigger the user to be
re-joined to a public room.
(With that said, although this sounds vaguely familiar in my lossy
memory, I can't find a ticket that actually describes this bug, so this
might be fictional. Edit: #16290 seems to describe this, although the
title is misleading.)
Additionally, as a side effect of the proposed fix (deleting the
`profiles` row), this PR also now deletes custom profile fields upon
user erasure, which is a new feature/bugfix (not sure which) in its own
right.
I do not see a ticket that corresponds to this feature gap, possibly
because custom profile fields are still a niche feature without
mainstream support (to the best of my knowledge).
Tests are included for the primary bugfix and for the cleanup of custom
profile fields.
### `set_displayname` module API change
This change includes a minor _technically_-breaking change to the module
API.
The change concerns `set_displayname` which is exposed to the module API
with a `deactivation: bool = False` flag, matching the internal handler
method it wraps.
I suspect that this is a mistake caused by overly-faithfully piping
through the args from the wrapped method (this Module API was introduced
in
https://github.com/matrix-org/synapse/pull/14629/changes#diff-0b449f6f95672437cf04f0b5512572b4a6a729d2759c438b7c206ea249619885R1592).
The linked PR did the same for `by_admin` originally before it was
changed.
The `deactivation` flag's only purpose is to be piped through to other
Module API callbacks when a module has registered to be notified about
profile changes.
My claim is that it makes no sense for the Module API to have this flag
because it is not the one doing the deactivation, thus it should never
be in a position to set this to `True`.
My proposed change keeps the flag (for function signature
compatibility), but turns it into a no-op (with a `ERROR` log when it's
set to True by the module).
The Module API callback notifying of the module-caused displayname
change will therefore now always have `deactivation = False`.
*Discussed in
[`#synapse-dev:matrix.org`](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$1f8N6G_EJUI_I_LvplnVAF2UFZTw_FzgsPfB6pbcPKk?via=element.io&via=matrix.org&via=beeper.com)*
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Summary
- drop the `systemd` extra from `pyproject.toml` and the
`systemd-python` optional dependency
- this means we don't ship the journald log handler, so it clarifies the
docs how to install that in the venv
- ensure the Debian virtualenv build keeps shipping
`systemd-python>=231` in the venv, so the packaged log config can keep
using `systemd.journal.JournalHandler`
Context of this is the following:
> Today in my 'how hard would it be to move to uv' journey:
https://github.com/systemd/python-systemd/issues/167
>
> The gist of it is that uv really wants to create a universal lock
file, which means it needs to be able to resolve the package metadata,
even for packages locked for other platforms. In the case of
systemd-python, they use mesonpy as build backend, which doesn't
implement prepare_metadata_for_build_wheel, which means it needs to run
meson to be able to resolve the package metadata. And it will hard-fail
if libsystemd dev headers aren't available 😭
>
> [*message in
#synapse-dev:matrix.org*](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$OKLB3TJVXAwq43sAZFJ-_PvMMzl4P_lWmSAtlmsoMuM?via=element.io&via=matrix.org&via=beeper.com)
- Update `synapse_xxx` (server-level) metrics to use
`server_name="$server_name",` instead of `instance="$instance"`
- Add `synapse_server_name_info` metric to map Synapse `server_name`s to
the `instance`s they're hosted on.
- For process level metrics, update to use `xxx * on (instance, job,
index) group_left(server_name)
synapse_server_name_info{server_name="$server_name"}`
All of the changes here are backwards compatible with whatever people
were doing before with their Prometheus/Grafana dashboards.
Previously, the recommendation was to use the `instance` label to group
everything under the same server (https://github.com/element-hq/synapse/blob/803e4b4d884b2de4b9e20dc47ffb59a983b8a4b5/docs/metrics-howto.md#L93-L147)
But the `instance` label actually has a special meaning and we're
actually abusing it by using it that way:
> `instance`: The `<host>:<port>` part of the target's URL that was
scraped.
>
> *--
https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series*
Since https://github.com/element-hq/synapse/issues/18592 (Synapse
`v1.139.0`), we now have the `server_name` label to use instead.
---
Additionally, the assumption that a single process is serving a single
server is no longer true with [Synapse Pro for small
hosts](https://docs.element.io/latest/element-server-suite-pro/synapse-pro-for-small-hosts/overview/).
Part of https://github.com/element-hq/synapse-small-hosts/issues/106
### Motivating use case
Although this change also benefits [Synapse Pro for small
hosts](https://docs.element.io/latest/element-server-suite-pro/synapse-pro-for-small-hosts/overview/)
(https://github.com/element-hq/synapse-small-hosts/issues/106), this is
actually spawning from adding Prometheus metrics to our workerized
Docker image (https://github.com/element-hq/synapse/pull/19324,
https://github.com/element-hq/synapse/pull/19336) with a more correct
label setup (without `instance`) and wanting the dashboard to be better.
### Testing strategy
1. Make sure your firewall allows the Docker containers to communicate
to the host (`host.docker.internal`) so they can access exposed ports of
other Docker containers. We want to allow Synapse to access the
Prometheus container and Grafana to access to the Prometheus container.
- `sudo ufw allow in on docker0 comment "Allow traffic from the default
Docker network to the host machine (host.docker.internal)"`
- `sudo ufw allow in on br-+ comment "(from Matrix Complement testing)
Allow traffic from custom Docker networks to the host machine
(host.docker.internal)"`
- [Complement firewall
docs](https://github.com/matrix-org/complement/blob/ee6acd9154bbae2d0071a9cb39593c0a5e37268b/README.md#potential-conflict-with-firewall-software)
1. Build the Docker image for Synapse: `docker build -t
matrixdotorg/synapse -f docker/Dockerfile .`
([docs](https://github.com/element-hq/synapse/blob/7a24fafbc376b9bffeb3277b1ad4aa950720c96c/docker/README-testing.md#building-and-running-the-images-manually))
1. Generate config for Synapse:
```
docker run -it --rm \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
-e SYNAPSE_REPORT_STATS=yes \
-e SYNAPSE_ENABLE_METRICS=1 \
matrixdotorg/synapse:latest generate
```
1. Start Synapse:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-p 8008:8008 \
-p 19090:19090 \
matrixdotorg/synapse:latest
```
1. You should be able to see metrics from Synapse at
http://localhost:19090/_synapse/metrics
1. Create a Prometheus config (`prometheus.yml`)
```yaml
global:
scrape_interval: 15s
scrape_timeout: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: prometheus
scrape_interval: 15s
metrics_path: /_synapse/metrics
scheme: http
static_configs:
- targets:
# This should point to the Synapse metrics listener (we're using
`host.docker.internal` because this is from within the Prometheus
container)
- host.docker.internal:19090
```
1. Start Prometheus (update the volume bind mount to the config you just
saved somewhere):
```
docker run \
--detach \
--name=prometheus \
--add-host host.docker.internal:host-gateway \
-p 9090:9090 \
-v
~/Documents/code/random/prometheus-config/prometheus.yml:/etc/prometheus/prometheus.yml
\
prom/prometheus
```
1. Make sure you're seeing some data in Prometheus. On
http://localhost:9090/query, search for `synapse_build_info`
1. Start [Grafana](https://hub.docker.com/r/grafana/grafana)
```
docker run -d --name=grafana --add-host
host.docker.internal:host-gateway -p 3000:3000 grafana/grafana
```
1. Visit the Grafana dashboard, http://localhost:3000/ (Credentials:
`admin`/`admin`)
1. **Connections** -> **Data Sources** -> **Add data source** ->
**Prometheus**
- Prometheus server URL: `http://host.docker.internal:9090`
1. Import the Synapse dashboard: `contrib/grafana/synapse.json`
To test workers, you can use the testing strategy from
https://github.com/element-hq/synapse/pull/19336 (assumes both changes
from this PR and the other PR are combined)
Fixes#19347
This deprecates MSC2697 which has been closed since May 2024. As per
#19347 this seems to be a thing we can just rip out. The crypto team
have moved onto MSC3814 and are suggesting that developers who rely on
MSC2697 should use MSC3814 instead.
MSC2697 implementation originally introduced by https://github.com/matrix-org/synapse/pull/8380
Spawning from wanting to [run a load
test](https://github.com/element-hq/synapse-rust-apps/pull/397) against
the Complement Docker image of Synapse and see metrics from the
homeserver.
### Why not just provide your own homeserver config?
Probably possible but it gets tricky when you try to use the workers
variant of the Docker image (`docker/Dockerfile-workers`). The way to
workaround it would probably be to `yq` edit everything in a script and
change `/data/homeserver.yaml` and `/conf/workers/*.yaml` to add the
`metrics` listener. And then modify `/conf/workers/shared.yaml` to add
`enable_metrics: true`. Doesn't spark much joy.
I just stumbled across the fact that my config used delegation as
recommended by the docs, and hosted Synapse on a subdomain. However my
config never had `public_baseurl` set and worked without issues, until I
just now tried to setup OIDC.
OIDC is initialized by the client instructing to open a URL on the
homeserver, and initially the correct URL is called, but Synapse does
not recognize it without `public_baseurl` being set correctly. After
changing this it immediately started working.
So in order to prevent anybody from making the same mistake, this adds a
small clarifying block in the OIDC docs.
