Fixes#20035
`_MultipartParserProtocol` assigned each `on_header_field` /
`on_header_value` / `on_part_data` value instead of accumulating it.
python-multipart calls these once per `dataReceived` chunk, so any
header name, header value or JSON part split across two chunks kept only
its last piece. A long `Location` in a federated media redirect was the
visible case: the truncated URL had no host and `get_file` failed with
`TypeError`.
The protocol now buffers header names, header values and the JSON part
in `bytearray`s and acts on them in `on_header_end` / `on_part_end`,
with a part counter replacing the `in_json` / `json_done` / `in_*`
flags. The file part still streams straight to the output stream, and
the existing `max_length` check still bounds everything that gets
buffered.
The new tests parse a file response and a redirect response split at
every byte offset, and byte by byte, and check the result matches the
unsplit parse. Before the fix, 100 and 115 split offsets respectively
produced a wrong result.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Signed-off-by: Ankit Jha <ankit.jha@tradomate.one>
Signed-off-by: Ankit Jha <jhaankit373@gmail.com>
`DeviceHandler.get_user_ids_changed`, which the sliding sync e2ee
extension and `GET /keys/changes` use, adds a member to
`newly_joined_or_invited_or_knocked_users` only when the previous
membership was none of join, invite or knock. A user going from invite
to join is skipped, so other members are never told to refresh the
devices of a user whose device list changed while they were only
invited. Classic sync does report them.
A change to join from any membership other than join is now counted.
join -> join (a profile change) is still ignored.
`test_extension_e2ee.py` covers invite -> join, invite -> join -> leave
in one window and a profile change, `test_keys.py` covers invite -> join
through `/keys/changes`. Not covered: device list changes made while a
user is only invited (#3504), which are still not announced until the
user joins.
Fixes#20486
Follows on from #20289 and #20433. Properly fixes handling of concurrent
requests (on single workers).
This tries to fix two cases:
1. We are *computing* responses for multiple syncs at the same time,
which currently causes the cache to get cross-contaminated.
2. Two syncs concurrent syncs on the wire (but don't overlap during
processing). When the second request is processed we correctly forget
any entries from the first request, however we currently still update
the cache with members returned on the second request. However, there is
no guarantee that the second request is what the client actually reads.
We use three rules to handle concurrent requests:
1. A response only uses cached members recorded at or before the
request's `since`. Members recorded after it cannot be from an ancestor
sync response.
2. Once a response has been built, we forget all already cached members
recorded after its `since`. This handles the client reusing an older
`since` token. Doing it at the end of the build rather than the start
also forgets members recorded by responses built at the same time.
3. We track the highest position seen for the device (`_newest`), from
the `since` of each request and the `next_batch` of each built response.
A response only records its members if its `next_batch` is after that
position. This handles responses that end at the same position, or that
finish out of order.
With these rules, when concurrent requests happen we throw away all
existing cached members and don't record any further members until one
sync has advanced sufficiently such that there are no longer any
concurrent sync requests.
Note that ideally we'd have some sort of per-connection identity so that
we can track this properly (like in sliding sync). However, that is a
bigger change. Instead, we note that we just need to detect concurrent
requests, and not update the cache in those cases.
Also note, can only detect concurrent requests that go to the same
worker.
Fixes#17480
Once `RemoteHandler` is connected, `whenConnected` returns a Deferred
that has already fired, so `writer` runs synchronously inside
`addCallbacks` and clears `_connection_waiter`. The next line then set
the waiter to that fired Deferred, so every later `_connect` call
returned early and logs stopped after the first line or two.
This sets the waiter before adding the callbacks, so a callback that
runs right away can still clear it. The new test logs a few more lines
over an existing connection and checks that they all arrive.
The bug came in with #9591, hence v1.30.0 in the changelog.
Thanks to @haarp for tracking this down and for the detailed
reproduction steps.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
`refresh_certificate` used to build a new `ServerContextFactory` on
every SIGHUP and then replace the factory on each listening port with a
fresh `TLSMemoryBIOFactory` wrapping the new one, reaching into the
port's `factory` attribute to do so.
Twisted asks the context factory for a context on every new connection,
so rebuilding the `SSL.Context` inside the existing factory is enough
for new connections to pick up the new certificate. `start()` now builds
the factory once, and `refresh_certificate` is only the SIGHUP handler.
This is spawning of #20452: Twisted 26.10 annotates `tcp.Port.factory`
as `Factory | None`, and mypy does not accept a `TLSMemoryBIOFactory`
there because its protocol type parameter does not match, so the old
assignment no longer type-checks. The change removes the assignment
rather than suppressing the error.
Verified with trial against Twisted 23.8.0 (our minimum declared dep),
26.4.0 (the current version in the lockfile) and 26.10.0 (the next
version), and with mypy against 26.4.0 and 26.10.0.
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.4 to 0.6.5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pyasn1/pyasn1/releases">pyasn1's
releases</a>.</em></p>
<blockquote>
<h2>Release 0.6.5</h2>
<h2>Revision 0.6.5, released 26-10-2026</h2>
<ul>
<li>GHSA-mm57-4v3p-rchx: Decoded binary REAL mantissas in linear time
instead of repeatedly shifting a growing integer and slicing the
remaining bytes (thanks for reporting, manus-use, manus-pi, ucsurf,
adriank31)</li>
<li>GHSA-2qm8-x737-gmw5: Reassembled constructed BIT STRING fragments
in linear time while preserving their decoded bit values, including
intermediate fragment padding (thanks for reporting, manus-use)</li>
<li>GHSA-9m75-vc3g-4x39: Reassembled constructed OCTET STRING and
character-string fragments and indefinite-length ANY content in
linear time. Nested constructed OCTET STRING fragments now contribute
their decoded octets (thanks for reporting, alimony, manus-pi,
adriank31)</li>
<li>GHSA-fm9c-5rcf-2q5p: Bounded schemaless container type tracking
after two distinct tag sets to prevent quadratic CPU use from
colliding long tag IDs (thanks for reporting, droideck)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst">pyasn1's
changelog</a>.</em></p>
<blockquote>
<h2>Revision 0.6.5, released 26-10-2026</h2>
<ul>
<li>GHSA-mm57-4v3p-rchx: Decoded binary REAL mantissas in linear time
instead of repeatedly shifting a growing integer and slicing the
remaining bytes (thanks for reporting, manus-use, manus-pi, ucsurf,
adriank31)</li>
<li>GHSA-2qm8-x737-gmw5: Reassembled constructed BIT STRING fragments
in linear time while preserving their decoded bit values, including
intermediate fragment padding (thanks for reporting, manus-use)</li>
<li>GHSA-9m75-vc3g-4x39: Reassembled constructed OCTET STRING and
character-string fragments and indefinite-length ANY content in
linear time. Nested constructed OCTET STRING fragments now contribute
their decoded octets (thanks for reporting, alimony, manus-pi,
adriank31)</li>
<li>GHSA-fm9c-5rcf-2q5p: Bounded schemaless container type tracking
after two distinct tag sets to prevent quadratic CPU use from
colliding long tag IDs (thanks for reporting, droideck)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pyasn1/pyasn1/commit/5d1673657b729fb580467e2cf4875856bbca5e59"><code>5d16736</code></a>
Prepare release 0.6.5</li>
<li><a
href="https://github.com/pyasn1/pyasn1/commit/a36e3a3d067148d26197a6b7221fd0e2cde9cc86"><code>a36e3a3</code></a>
Fix quadratic CPU use in schemaless container decoding</li>
<li><a
href="https://github.com/pyasn1/pyasn1/commit/4ce00797c594b59c1fd6b86d9ebc3ba87b7ec9f9"><code>4ce0079</code></a>
Remove quadratic constructed OCTET STRING and ANY reassembly</li>
<li><a
href="https://github.com/pyasn1/pyasn1/commit/73ba7b043cc7e92cb37f16f2c955bff93b0e6f6c"><code>73ba7b0</code></a>
Remove quadratic constructed BIT STRING reassembly</li>
<li><a
href="https://github.com/pyasn1/pyasn1/commit/ce3ec3390784780a530047e0e77be5ff457636a4"><code>ce3ec33</code></a>
Remove quadratic binary REAL mantissa decoding</li>
<li>See full diff in <a
href="https://github.com/pyasn1/pyasn1/compare/v0.6.4...v0.6.5">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/element-hq/synapse/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reloading the log config on SIGHUP writes every record back into
Twisted's logging. `redirect_stdio_to_logs()` replaces `sys.stderr` with
a `LoggingFile` after startup, and `_load_logging_config` then rebuilds
`StreamHandler`s on it, so records hit the `logging during logging`
guard in `one_time_logging_setup` and JSON lines get wrapped a second
time.
`dictConfig` now runs with `sys.__stdout__` and `sys.__stderr__` swapped
in, and the redirected streams are put back afterwards, including when
the config is invalid. `tests/config/test_logger.py` covers both. Not
covered: Windows and configs that use a `FileHandler`.
Fixes#20091
### Pull Request Checklist
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (ruff check, ruff format and mypy run on the changed files)
Adds a `synapse_thread_cpu_seconds_total{thread_type, mode}` metric,
grouping threads into:
1. the `main` thread
2. each registered threadpool (`threadpool_*`)
3. `tokio`
4. `other`
The kernel only reports CPU time for live threads, so the collector sums
per-thread deltas between scrapes. A thread that exits loses the CPU it
used since the last scrape.
This is primarily useful for seeing how busy the main thread is vs other
ones.
Fixes#18885.
The "Override ratelimiting for users" section of the user admin API docs
described `messages_per_second` as "the number of actions that can be
performed in a second", which suggests the override applies to every
ratelimiter. In the code (`synapse/api/ratelimiting.py`) the override's
rate/burst values are only used by the event-sending ratelimiter
(`RequestRatelimiter.ratelimit`, i.e. `rc_message`); the generic
`Ratelimiter.can_do_action` only reads the override to check whether
`messages_per_second` is `0` and, if so, disables ratelimiting entirely
for that user.
Changes (docs only):
- State up front that the override replaces only the message-sending
limits (`rc_message`), not `rc_invites`/`rc_joins`/`rc_login` etc., and
that `messages_per_second = 0` disables all ratelimiting.
- Rename the sub-headings to "Get status of / Set / Delete ratelimit
**override**" and describe the fields as messages rather than actions.
- Explain that "Delete ratelimit override" restores the global config
limits rather than disabling ratelimiting.
The `#override-ratelimiting-for-users` anchor referenced elsewhere in
the page is unchanged.
Verified with `mdbook build` (mdbook 0.5.2, same as `docs-pr.yaml`):
renders cleanly, new anchors `#get-status-of-ratelimit-override`,
`#set-ratelimit-override`, `#delete-ratelimit-override` present.
### Pull Request Checklist
A notary key query that leaves out minimum_valid_until_ts is rejected
with errcode M_MISSING_PARAM and an error naming that field as required.
Callers that follow the server-server API, where the timestamp is
optional and the server is supposed to use the current time, cannot
complete the query. The criteria model typed minimum_valid_until_ts as
an integer or null but gave it no default, so validation treated the
field as required and failed before the query ran.
An explicit null was already a separate accepted value, used for
half-lifetime cache behaviour, and was not the same as omitting the
field.
The field now defaults to null, so an omitted timestamp validates. After
parsing, each key whose criteria did not include the field is replaced
with a copy that sets minimum_valid_until_ts to the current time in
milliseconds. Explicit null is left unchanged, and an empty key-id map
is left unchanged so a request for every cached key still ignores
validity. Tests cover the omitted timestamp against cache lifetime,
mixed criteria including zero and null, non-integer timestamps, and a
body without server_keys, and a bugfix newsfragment records the default.
Ran `python3 -m pytest -q tests/rest/key/v2/test_remote_key_resource.py`
in a Linux container (`python:3.12`) for the synapse RemoteKey change; 7
tests passed.
The tests in `tests/rest/key/v2/test_remote_key_resource.py` fail
without the fix in the synapse RemoteKey change (checked by reverting
the source changes).
Fixes#20466
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This PR should be reviewable on a commit-by-commit basis. If you would
prefer that it is broken down into multiple PRs then I can do so.
Fixes to match latest MSC4388:
- Return `200` with `create_available=false` for `GET` when configured
in `authenticated` mode and no access token presented
- Return `401` for a `GET` where an invalid/expired token is presented
- Return `404` for `DELETE` where session has expired but not yet
evicted
Features:
- Rate limiting which is configurable via
`experimental_features.msc4388_rc_create` (for `POST` and is per
requester for authenticated requests or per client IP for
unauthenticated requests) and
`experimental_features.msc4388_rc_requests` (for `GET`/`PUT`/`DELETE`).
- Allow the session TTL to be configured via the
`experimental_features.msc4388_session_ttl`. As per spec this is between
2 and 5 minutes.
- Concurrent sessions are unlimited for authenticated requesters
(subject to the rate limits).
- Unauthenticated requesters have a maximum concurrent sessions limit
(default 1,000) which is configurable by
`experimental_features.msc4388_max_unauthenticated_sessions`.
Misc:
- Code and comment cleanup
The experimental feature is tracked by
https://github.com/element-hq/synapse/issues/19433 and I will update it
with the new config options when this PR merges.
Fix a bug introduced in Synapse v1.146.0 where failed remote media
downloads and URL previews would leave behind empty files in the media
store.
Regressed in #19204 which restructured
`MediaStorage.store_into_file(...)` to support running without a local
media store. As part of that, the `yield` (where the caller writes into
the file) was moved outside of the `try`/`except` that cleans up the
file on failure. Since `store_into_file(...)` is an
`asynccontextmanager`, an exception raised by the caller is thrown back
in at the `yield`, so it now bypasses the cleanup entirely.
This means any failed remote media download leaves a zero-byte file in
`remote_content/` (and similarly `url_cache/` for URL previews) without
a corresponding database row. Most of these fail before a single byte is
written, for example `NotRetryingDestination` while a remote server is
backed off, which fails instantly without any network request. Media
retention works from the database so it never cleans these up, and
because each attempt uses a fresh `filesystem_id`, clients repeatedly
requesting media (e.g. avatars of members on dead servers) keep
accumulating new ones. On a small homeserver, this added up to ~6000
orphaned files (and ~36000 directories) since upgrading to v1.146.0.
This PR moves the `yield` back inside the `try` (as it was before
#19204) so the file is removed again on failure. This applies to both
the local media store and the temporary file used when
`enable_local_media_storage` is disabled.
### Dev notes
To find orphaned files on an affected homeserver (zero-byte files with
no matching `remote_media_cache` row):
```shell
find /path/to/media_store/remote_content /path/to/media_store/url_cache -type f -size 0
```
Any of these without a matching row in `remote_media_cache` (or
`local_media_repository` for `url_cache/`) are orphans and safe to
delete, along with any empty directories left behind.
`KeyStore.get_server_keys_json` orders the `server_keys_json` rows of a
key by `ts_added_ms` and each row overwrites the previous one, so the
last notary to answer wins even when its `ts_valid_until_ms` is lower
than an earlier answer's. When that answer is stale, `StoreKeyFetcher`
returns a key below `minimum_valid_until_ts` and event signature
verification fails although a long enough key is stored.
The query now orders by `ts_valid_until_ms, ts_added_ms`, so the
longest-valid entry wins and the most recent one still wins a tie.
`tests/crypto/test_keyring.py` stores two notary responses for one key,
the longer-lived one first, and checks that the lookup and
`verify_json_for_server` use it. Not covered:
`get_server_keys_json_for_remote` and
`get_all_server_keys_json_for_remote`, which still prefer the most
recently added row.
Fixes#20358
### Pull Request Checklist
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (ruff check, ruff format and mypy run on the changed files)
Follow-up to https://github.com/element-hq/synapse/pull/20407, which
added the `excluded_user_ids` option without enforcing it. This makes
legacy `/sync` respect it: users listed in a room's `excluded_user_ids`
no longer get that room through the synthetic join. They can still read
it, as it is world-readable.
The cached lookup now maps room ID to config ID, so the per-user filter
runs in memory and adds no database query to `/sync`.
`get_server_announcement_room_ids()` is kept as a wrapper, so the
stacked PRs are unaffected.
Known limitation, documented in the config docs: changing the list does
not update rooms that clients already have. A newly excluded user keeps
the room, which stops updating. A user removed from the list gets new
events without the room's state until their next initial sync. I think
that's an okay limitation for now, as we don't really expect this list
to be used for real users
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
> [!NOTE]
> Epic: [Efficient Server
Announcements](https://github.com/element-hq/synapse/issues/20240)
> Follows: https://github.com/element-hq/synapse/pull/20372
This PR first tests that every user can read the server announcements
through the "history" endpoints like
- [x] `GET /rooms/{roomId}/messages`
- [x] `GET /rooms/{roomId}/context/{eventId}`
- [x] `GET /rooms/{roomId}/event/{eventId}`
- [x] `GET /rooms/{roomId}/relations/{eventId}`
- [x] `GET /rooms/{roomId}/threads`
- [x] `GET /rooms/{roomId}/initialSync`
This already works for users who have never been in the room, because
announcement rooms are world-readable.
It doesn't for a sender who was removed from the room's config: Synapse
still uses their `leave` membership and shows them the room as it was
when they left.
This PR then treats a user who left a configured announcement room like
a user who has never been in it.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Quentin Gliech <quenting@element.io>
Andrew Morgan