MSC4430 identifies room members by a per-user, per-room Ed25519 "member key"
rather than a user ID, and lets a member's `m.room.member` event carry the user
ID it claims to be (`content.localpart`, `content.domain`), co-signed by the
claimed domain's server signing key. This adds the two tables and the store
behind that, with no callers: nothing in the server creates keys or learns links
yet, and the member key room version cannot be used.
`msc4430_member_keys` holds the keys we hold for our own users, one per (local
user, room), with the private seed. `MemberKeysWorkerStore.create_local_member_key`
is insert-if-absent and returns whichever key is stored, so a user's key in a
room can never change once it exists, however many concurrent callers try (the
earlier monolithic attempt raised `IntegrityError` here). The store derives the
public form from the signing key it is given, so the two columns cannot
disagree, and rejects any key ID other than `ed25519:1`. Every lookup by key
takes the room the key was seen in: a key of ours from room A appearing in room
B's events does not resolve there (the earlier attempt mapped keys without a
room and gave users phantom memberships). Lookups are cached per (room, user),
per (room, key), per user and per room, and `create_local_member_key`
invalidates all four on the cache stream, so workers see new keys.
`msc4430_member_links` holds what we have learned about keys we do not hold:
the claimed domain (routing information, kept whether verified or not), the
claimed localpart (NULL once the event that carried it is redacted, after
which the key maps to no user), and whether the link is verified. A verified
link always names the `m.room.member` event and the server signing key ID that
proved it; the `MemberLink` type and a CHECK constraint both enforce this, so
"verified" means exactly one thing. Our own keys get no link row: a link is a
claim someone else made, and the keys we hold are already known to be ours.
`upsert_member_link` is a plain replace that returns the previously stored link,
so a caller can see what changed (the mapped user, the domain); which events
may update a link is the caller's policy, not the store's, and
`get_member_link_txn` plus `upsert_member_link_txn` let it read, decide and
write in one transaction. `mark_links_unverified(domain, key_id)` downgrades
every link a rotated or withdrawn server key vouched for, in one bulk cache
invalidation, keeping the event and key ID so the links can be found again
(`get_links_for_signing_key`) if the events are re-signed; a key ID alone does
not identify a key, so re-checking a link means re-checking the co-signature
on its event against the domain's current key material, which the schema
says. `get_member_keys_for_domain_in_room(verified_only=...)` separates routing
(claimed) from trust (verified), and answers for our own server name from the
keys we hold, never from links: a link claiming our domain for a key we do not
hold is somebody else's claim. `get_member_links_for_claimed_user` is the
reverse mapping; it returns every claimant (verified first) rather than picking
one, because an unverified claim is anyone's to make and that lookup is how a
user ID in an invite, kick or power level change becomes a key.
`clear_link_localpart_for_redacted_event_txn` is the erasure hook (it clears
every link learned from the redacted event). Purging a room clears its links;
the keys we hold deliberately survive a purge, since a user's key in a room
must not change if the room is re-joined. The stored seed cannot be revoked
(MSC4430 has no member key rotation; only rotating the server key helps), and
the schema says so.
The store is mixed into `RoomMemberWorkerStore` (so it is present on workers)
rather than registered on `DataStore`, because the membership queries are the
code that will need it. `LocalMemberKey` and `MemberLink` are frozen attrs
types in `synapse/storage/member_keys.py`, next to the other storage return
types; `SigningKey` is added to the mypy plugin's immutable types so a
`LocalMemberKey` can be a cached value. `synapse_port_db` learns the boolean
column. `MemberKeyHandler.is_local_identity` (and so `hs.is_mine_identity`) now
answers for member keys by looking the key up in its room, instead of raising.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fixes#19375
`prometheus_client` 0.24 makes `Collector` a generic type.
Previously, `InFlightGauge` inherited from both `Generic[MetricsEntry]`
and `Collector`, resulting in the error `TypeError: cannot create a
consistent MRO` when using `prometheus_client` >= 0.24. This behaviour
of disallowing multiple `Generic` inheritance is more strictly enforced
starting with python 3.14, but can still lead to issues with earlier
versions of python.
This PR separates runtime and typing inheritance for `InFlightGauge`:
- Runtime: `InFlightGauge` inherits only from `Collector`
- Typing: `InFlightGauge` is generic
This preserves static typing, avoids MRO conflicts, and supports both
`prometheus_client` <0.24 and >=0.24.
I have tested these changes out locally with `prometheus_client` 0.23.1
& 0.24 on python 3.14 while sending a bunch of messages over federation
and watching a grafana dashboard configured to show
`synapse_util_metrics_block_in_flight_total` &
`synapse_util_metrics_block_in_flight_real_time_sum` (the only metric
setup to use `InFlightGauge`) and things are working in each case.
https://github.com/element-hq/synapse/blob/a1e9abc7df3e6c43a95cba059348546a4c9d4491/synapse/util/metrics.py#L112-L119
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Introduce `Clock.add_system_event_trigger(...)` to wrap system event
callback code in a logcontext, ensuring we can identify which server
generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.add_system_event_trigger(...)` over
`reactor.addSystemEventTrigger(...)`
Part of https://github.com/element-hq/synapse/issues/18905
Introduce `Clock.call_when_running(...)` to wrap startup code in a
logcontext, ensuring we can identify which server generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.call_when_running(...)` over
`reactor.callWhenRunning(...)`
Part of https://github.com/element-hq/synapse/issues/18905
Bulk refactor `Histogram` metrics to be homeserver-scoped. We also add
lints to make sure that new `Histogram` metrics don't sneak in without
using the `server_name` label (`SERVER_NAME_LABEL`).
Part of https://github.com/element-hq/synapse/issues/18592
### Testing strategy
1. Add the `metrics` listener in your `homeserver.yaml`
```yaml
listeners:
# This is just showing how to configure metrics either way
#
# `http` `metrics` resource
- port: 9322
type: http
bind_addresses: ['127.0.0.1']
resources:
- names: [metrics]
compress: false
# `metrics` listener
- port: 9323
type: metrics
bind_addresses: ['127.0.0.1']
```
1. Start the homeserver: `poetry run synapse_homeserver --config-path
homeserver.yaml`
1. Fetch `http://localhost:9322/_synapse/metrics` and/or
`http://localhost:9323/metrics`
1. Observe response includes the TODO metrics with the `server_name`
label
### Todo
- [x] Wait for https://github.com/element-hq/synapse/pull/18656 to merge
### Dev notes
```
LoggingDatabaseConnection
make_conn
make_pool
make_fake_db_pool
```
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Bulk refactor `Gauge` metrics to be homeserver-scoped. We also add lints
to make sure that new `Gauge` metrics don't sneak in without using the
`server_name` label (`SERVER_NAME_LABEL`).
Part of https://github.com/element-hq/synapse/issues/18592
### Testing strategy
1. Add the `metrics` listener in your `homeserver.yaml`
```yaml
listeners:
# This is just showing how to configure metrics either way
#
# `http` `metrics` resource
- port: 9322
type: http
bind_addresses: ['127.0.0.1']
resources:
- names: [metrics]
compress: false
# `metrics` listener
- port: 9323
type: metrics
bind_addresses: ['127.0.0.1']
```
1. Start the homeserver: `poetry run synapse_homeserver --config-path
homeserver.yaml`
1. Fetch `http://localhost:9322/_synapse/metrics` and/or
`http://localhost:9323/metrics`
1. Observe response includes the TODO metrics with the `server_name`
label
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Bulk refactor `Counter` metrics to be homeserver-scoped. We also add
lints to make sure that new `Counter` metrics don't sneak in without
using the `server_name` label (`SERVER_NAME_LABEL`).
All of the "Fill in" commits are just bulk refactor.
Part of https://github.com/element-hq/synapse/issues/18592
### Testing strategy
1. Add the `metrics` listener in your `homeserver.yaml`
```yaml
listeners:
# This is just showing how to configure metrics either way
#
# `http` `metrics` resource
- port: 9322
type: http
bind_addresses: ['127.0.0.1']
resources:
- names: [metrics]
compress: false
# `metrics` listener
- port: 9323
type: metrics
bind_addresses: ['127.0.0.1']
```
1. Start the homeserver: `poetry run synapse_homeserver --config-path
homeserver.yaml`
1. Fetch `http://localhost:9322/_synapse/metrics` and/or
`http://localhost:9323/metrics`
1. Observe response includes the `synapse_user_registrations_total`,
`synapse_http_server_response_count_total`, etc metrics with the
`server_name` label
This is so that we can cache it.
We also move the sliding sync types to
`synapse/types/handlers/sliding_sync.py`. This is mainly in-prep for
#17599 to avoid circular imports.
The only change in behaviour is that
`RoomSyncConfig.combine_sync_config(..)` now returns a new room sync
config rather than mutating in-place.
Reviewable commit-by-commit.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
Co-authored-by: David Robertson <davidr@element.io>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
Assert that the return type of callables wrapped in @cached
and @cachedList are cachable (aka immutable).
The cached decorators always return a Deferred, which was not
properly propagated. It was close enough when wrapping coroutines,
but failed if a bare function was wrapped.
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
