ac26cfac83
Let me give you a clear summary of where this project stands:
...
---
Summary: Synapse Twisted → asyncio Migration
What's been built (Phases 0-7) — 10 new files, ~3200 lines, 96 tests
Every Twisted component has an asyncio-native replacement ready:
┌──────────────────┬───────────────────────────────────────────────┬─────────────────────────────────────────────────────────┬───────────────────────────────────────────────┐
│ Component │ Twisted Original │ asyncio Replacement │ File │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ LoggingContext │ threading.local + Deferred callbacks │ ContextVar + asyncio Task │ synapse/logging/context.py (native functions) │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Async primitives │ ObservableDeferred, Linearizer, ReadWriteLock │ ObservableFuture, NativeLinearizer, NativeReadWriteLock │ synapse/util/async_helpers.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Cache │ DeferredCache │ FutureCache │ synapse/util/caches/future_cache.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Clock │ Clock (Twisted reactor) │ NativeClock (asyncio) │ synapse/util/clock.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Database │ adbapi.ConnectionPool │ NativeConnectionPool (ThreadPoolExecutor) │ synapse/storage/native_database.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ HTTP Client │ treq + Twisted Agent │ aiohttp.ClientSession │ synapse/http/native_client.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ HTTP Server │ JsonResource + Twisted Site │ NativeJsonResource + aiohttp.web │ synapse/http/native_server.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Replication │ LineOnlyReceiver (Twisted Protocol) │ asyncio.StreamReader/Writer │ synapse/replication/tcp/native_protocol.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Event loop │ ISynapseReactor │ ISynapseEventLoop │ synapse/types/__init__.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Test helper │ — │ FakeAsyncioLoop │ tests/async_helpers.py │
└──────────────────┴───────────────────────────────────────────────┴─────────────────────────────────────────────────────────┴───────────────────────────────────────────────┘
What's been wired in safely — 224 files changed, 0 regressions
- MemoryReactor type hint → Any across 198 test files (cosmetic)
- synapse/http/server.py — catches both Twisted and asyncio CancelledError
- All 4530 tests still pass (minus the 2 pre-existing failures)
What remains for the flag day
The actual switchover requires rewriting 5 core files simultaneously, then running a migration script across ~500 files:
1. tests/unittest.py + tests/server.py — switch from twisted.trial.TestCase to unittest.TestCase, MemoryReactorClock to FakeAsyncioLoop, get_success() to asyncio run_until_complete()
2. synapse/logging/context.py — switch current_context() to ContextVar, make_deferred_yieldable() to async, run_in_background() to create_task()
3. synapse/util/async_helpers.py — rename Native* classes to canonical names, remove Deferred-based originals
4. Migration script — update all CancelledError, defer.*, Deferred imports across ~500 files
5. pyproject.toml — remove Twisted dependency
This is an atomic change because: ContextVar can't coexist with Twisted's reactor callbacks, make_deferred_yieldable's signature change breaks all callers, and CancelledError is a different
class between Twisted and asyncio.
2026-03-21 16:17:04 +00:00
fcac7e0282
Write union types as X | Y where possible ( #19111 )
...
aka PEP 604, added in Python 3.10
2025-11-06 14:02:33 -06:00
fc244bb592
Use type hinting generics in standard collections ( #19046 )
...
aka PEP 585, added in Python 3.9
- https://peps.python.org/pep-0585/
- https://docs.astral.sh/ruff/rules/non-pep585-annotation/
2025-10-22 16:48:19 -05:00
5a9ca1e3d9
Introduce Clock.call_when_running(...) to include logcontext by default ( #18944 )
...
Introduce `Clock.call_when_running(...)` to wrap startup code in a
logcontext, ensuring we can identify which server generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.call_when_running(...)` over
`reactor.callWhenRunning(...)`
Part of https://github.com/element-hq/synapse/issues/18905
2025-09-22 10:27:59 -05:00
6514381b02
Implement the push rules for experimental MSC4306: Thread Subscriptions. ( #18762 )
...
Follows: #18756
Implements: MSC4306
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org >
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com >
2025-08-06 15:33:52 +01:00
a31d53b28f
Use twisted.internet.testing module in tests instead of deprecated twisted.test.proto_helpers. ( #18728 )
...
Follows: #18727
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org >
2025-07-30 12:32:10 +01:00
88785dbaeb
Refactor cache metrics to be homeserver-scoped ( #18604 )
...
(add `server_name` label to cache metrics).
Part of https://github.com/element-hq/synapse/issues/18592
2025-07-16 16:04:57 -05:00
9512b84a72
Bump mypy from 1.10.1 to 1.11.2 ( #17842 )
...
Bumps [mypy](https://github.com/python/mypy ) from 1.10.1 to 1.11.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/python/mypy/blob/master/CHANGELOG.md ">mypy's
changelog</a>.</em></p>
<blockquote>
<h3>Mypy 1.11.2</h3>
<ul>
<li>Alternative fix for a union-like literal string (Ivan Levkivskyi, PR
<a
href="https://redirect.github.com/python/mypy/pull/17639 ">17639</a>)</li>
<li>Unwrap <code>TypedDict</code> item types before storing (Ivan
Levkivskyi, PR <a
href="https://redirect.github.com/python/mypy/pull/17640 ">17640</a>)</li>
</ul>
<h3>Acknowledgements</h3>
<p>Thanks to all mypy contributors who contributed to this release:</p>
<ul>
<li>Alex Waygood</li>
<li>Alexander Leopold Shon</li>
<li>Ali Hamdan</li>
<li>Anders Kaseorg</li>
<li>Ben Brown</li>
<li>Bénédikt Tran</li>
<li>bzoracler</li>
<li>Christoph Tyralla</li>
<li>Christopher Barber</li>
<li>dexterkennedy</li>
<li>gilesgc</li>
<li>GiorgosPapoutsakis</li>
<li>Ivan Levkivskyi</li>
<li>Jelle Zijlstra</li>
<li>Jukka Lehtosalo</li>
<li>Marc Mueller</li>
<li>Matthieu Devlin</li>
<li>Michael R. Crusoe</li>
<li>Nikita Sobolev</li>
<li>Seo Sanghyeon</li>
<li>Shantanu</li>
<li>sobolevn</li>
<li>Steven Troxler</li>
<li>Tadeu Manoel</li>
<li>Tamir Duberstein</li>
<li>Tushar Sadhwani</li>
<li>urnest</li>
<li>Valentin Stanciu</li>
</ul>
<p>I’d also like to thank my employer, Dropbox, for supporting mypy
development.</p>
<h2>Mypy 1.10</h2>
<p>We’ve just uploaded mypy 1.10 to the Python Package Index (<a
href="https://pypi.org/project/mypy/ ">PyPI</a>). Mypy is a static type
checker for Python. This release includes new features, performance
improvements and bug fixes. You can install it as follows:</p>
<pre><code>python3 -m pip install -U mypy
</code></pre>
<p>You can read the full documentation for this release on <a
href="http://mypy.readthedocs.io ">Read the Docs</a>.</p>
<h3>Support TypeIs (PEP 742)</h3>
<p>Mypy now supports <code>TypeIs</code> (<a
href="https://peps.python.org/pep-0742/ ">PEP 742</a>), which allows</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/python/mypy/commit/789f02c83a5d5cb35f5e33ba91df46c8fea6b28e "><code>789f02c</code></a>
Bump version to 1.11.2</li>
<li><a
href="https://github.com/python/mypy/commit/917cc75fd6f1417edb45eb77e449934f794c18fc "><code>917cc75</code></a>
An alternative fix for a union-like literal string (<a
href="https://redirect.github.com/python/mypy/issues/17639 ">#17639</a>)</li>
<li><a
href="https://github.com/python/mypy/commit/7d805b364ee80396e0b9ca906f32f901b2ac7e12 "><code>7d805b3</code></a>
Unwrap TypedDict item types before storing (<a
href="https://redirect.github.com/python/mypy/issues/17640 ">#17640</a>)</li>
<li><a
href="https://github.com/python/mypy/commit/32675dddfacccef616557916cb872757605ab493 "><code>32675dd</code></a>
Revert "Fix Literal strings containing pipe characters" (<a
href="https://redirect.github.com/python/mypy/issues/17638 ">#17638</a>)</li>
<li><a
href="https://github.com/python/mypy/commit/778542b93a6f5b3c168a8acc03717700ae6f8048 "><code>778542b</code></a>
Revert "Fix <code>RawExpressionType.accept</code> crash with
<code>--cache-fine-grained</code>" (<a
href="https://redirect.github.com/python/mypy/issues/1 ">#1</a>...</li>
<li><a
href="https://github.com/python/mypy/commit/14ab742dec6b58a4e94772115cb3b5c67a4b3d33 "><code>14ab742</code></a>
Bump version to 1.11.2+dev</li>
<li><a
href="https://github.com/python/mypy/commit/570b90a7a368f04c64f60af339d0ac1808c49c15 "><code>570b90a</code></a>
Bump version to 1.11</li>
<li><a
href="https://github.com/python/mypy/commit/b3a102ef31f63a8a8ba32c8dbe160ddef3c43054 "><code>b3a102e</code></a>
Fix <code>RawExpressionType.accept</code> crash with
<code>--cache-fine-grained</code> (<a
href="https://redirect.github.com/python/mypy/issues/17588 ">#17588</a>)</li>
<li><a
href="https://github.com/python/mypy/commit/aec04c74488d46a81a95ed3553b8e953a6ec59a7 "><code>aec04c7</code></a>
Fix PEP 604 isinstance caching (<a
href="https://redirect.github.com/python/mypy/issues/17563 ">#17563</a>)</li>
<li><a
href="https://github.com/python/mypy/commit/cb44e4d8f18b9bc874f1076b33eec7ad67de165c "><code>cb44e4d</code></a>
Fix <code>typing.TypeAliasType</code> being undefined on python <
3.12 (<a
href="https://redirect.github.com/python/mypy/issues/17558 ">#17558</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/python/mypy/compare/v1.10.1...v1.11.2 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-17 15:05:00 +00:00
11bc9a1b3a
Implement MSC4210: Remove legacy mentions ( #17783 )
2024-10-14 14:24:28 +01:00
23740eaa3d
Correctly mention previous copyright ( #16820 )
...
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
8e1e62c9e0
Update license headers
2023-11-21 15:29:58 -05:00
3b0083c92a
Use immutabledict instead of frozendict ( #15113 )
...
Additionally:
* Consistently use `freeze()` in test
---------
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com >
Co-authored-by: 6543 <6543@obermui.de >
2023-03-22 17:15:34 +00:00
20ed8c926b
Stabilize support for MSC3873: disambuguated event push keys. ( #15190 )
...
This removes the experimental configuration option and
always escapes the push rule condition keys.
Also escapes any (experimental) push rule condition keys
in the base rules which contain dot in a field name.
2023-03-07 11:27:57 -05:00
820f02b70b
Stabilize support for MSC3966: event_property_contains push condition. ( #15187 )
...
This removes the configuration flag & updates the identifiers to
use the stable version.
2023-03-07 10:06:02 -05:00
fd9cadcf53
Stabilize support for MSC3758: event_property_is push condition ( #15185 )
...
This removes the configuration flag & updates the identifiers to
use the stable version.
2023-03-06 08:38:01 -05:00
8ef324ea6f
Update intentional mentions (MSC3952) to depend on exact_event_property_contains (MSC3966). ( #15051 )
...
This replaces the specific `is_user_mention` push rule condition
used in MSC3952 with the generic `exact_event_property_contains`
push rule condition from MSC3966.
2023-03-02 08:30:51 -05:00
e746f80b4f
Do not accept pattern_type from user input in push rules. ( #15088 )
...
Internally the push rules module uses a `pattern_type` property for `event_match`
conditions (and `related_event_match`) to mark the condition as matching the
current user's Matrix ID or localpart.
This is leaky to the Client-Server API where a user can successfully set a condition
which provides `pattern_type` instead of `pattern` (note that there's no benefit to
doing this -- the user can just use their own Matrix ID or localpart instead). When
serializing back to the client the `pattern_type` property is converted into a proper
`pattern`.
The following changes are made to avoid this:
* Separate the `KnownCondition::EventMatch` enum value into `EventMatch`
and `EventMatchType`, each with their own expected properties. (Note that a
similar change is made for `RelatedEventMatch`.)
* Make it such that the `pattern_type` variants serialize to the same condition kind,
but cannot be deserialized (since they're only provided by base rules).
* As a final tweak, convert `user_id` vs. `user_localpart` values into an enum.
2023-02-28 10:11:20 -05:00
ec79870f14
Fix a typo in MSC3873 config option. ( #15138 )
...
Previously the experimental configuration option referred to the wrong
MSC number.
2023-02-23 16:06:42 -05:00
979f237b28
Update intentional mentions (MSC3952) to depend on exact_event_match (MSC3758). ( #15037 )
...
This replaces the specific `is_room_mention` push rule condition
used in MSC3952 with the generic `exact_event_match` push rule
condition from MSC3758.
No functionality changes due to this.
2023-02-16 09:51:22 -05:00
119e0795a5
Implement MSC3966: Add a push rule condition to search for a value in an array. ( #15045 )
...
The `exact_event_property_contains` condition can be used to
search for a value inside of an array.
2023-02-14 14:02:19 -05:00
14be78d492
Support for MSC3758: exact_event_match push condition ( #14964 )
...
This specifies to search for an exact value match, instead of
string globbing. It only works across non-compound JSON values
(null, boolean, integer, and strings).
2023-02-10 12:37:07 -05:00
c951fbedcb
MSC3873: Escape keys when flattening dicts. ( #15004 )
...
This disambiguates keys which attempt to match fields
with a dot in them (e.g. m.relates_to).
Disabled by default behind an experimental configuration flag.
2023-02-08 13:09:41 -05:00
5b55c32d61
Add tests for using _flatten_dict with an event. ( #15002 )
2023-02-07 06:56:09 -05:00
f0cae26d58
Add a docstring & tests for _flatten_dict. ( #14981 )
2023-02-03 16:48:13 +00:00
52700a0bcf
Support the backwards compatibility features in MSC3952. ( #14958 )
...
If the feature is enabled and the event has a `m.mentions` property,
skip processing of the legacy mentions rules.
2023-02-03 16:28:20 +00:00
2a51f3ec36
Implement MSC3952: Intentional mentions ( #14823 )
...
MSC3952 defines push rules which searches for mentions in a list of
Matrix IDs in the event body, instead of searching the entire event
body for display name / local part.
This is implemented behind an experimental configuration flag and
does not yet implement the backwards compatibility pieces of the MSC.
2023-01-27 10:16:21 -05:00
7f2cabf271
Fix-up type hints for tests.push module. ( #14816 )
2023-01-11 07:35:40 -05:00
3da6450327
Initial support for MSC3931: Room version push rule feature flags ( #14520 )
...
* Add support for MSC3931: Room Version Supports push rule condition
* Create experimental flag for future work, and use it to gate MSC3931
* Changelog entry
2022-11-28 16:29:53 -07:00
2d0ba3f89a
Implementation for MSC3664: Pushrules for relations ( #11804 )
2022-10-25 14:38:01 +01:00
09be8ab5f9
Remove the experimental implementation of MSC3772. ( #14094 )
...
MSC3772 has been abandoned.
2022-10-12 06:26:39 -04:00
285b9e9b6c
Speed up calculating push actions in large rooms ( #13973 )
...
We move the expensive check of visibility to after calculating push actions, avoiding the expensive check for users who won't get pushed anyway.
I think this should have a big impact on rooms with large numbers of local users that have pushed disabled.
2022-09-30 14:27:00 +01:00
ebd9e2dac6
Implement push rule evaluation in Rust. ( #13838 )
2022-09-29 16:12:09 +01:00
b4ae3b0d44
Don't include appservice users when calculating push rules ( #13332 )
...
This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078
2022-07-20 12:06:13 +01:00
88ce3080d4
Experimental support for MSC3772 ( #12740 )
...
Implements the following behind an experimental configuration flag:
* A new push rule kind for mutually related events.
* A new default push rule (`.m.rule.thread_reply`) under an unstable prefix.
This is missing part of MSC3772:
* The `.m.rule.thread_reply_to_me` push rule, this depends on MSC3664 / #11804 .
2022-05-24 13:23:23 +00:00
9d21ecf7ce
Add type hints to tests files. ( #12256 )
2022-03-21 09:43:16 -04:00
6c0b44a3d7
Fix PushRuleEvaluator and Filter to work on frozendicts ( #12100 )
...
* Fix `PushRuleEvaluator` to work on frozendicts
frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.
* Fix event filtering for frozen events
Looks like this one was introduced by #11194 .
2022-02-28 17:40:24 +00:00
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
...
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io >
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io >
Co-authored-by: Dan Callahan <danc@element.io >
2021-05-11 11:47:23 +02:00
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl >`
2021-04-14 15:34:27 +01:00
6f7417c3db
Handle missing content keys when calculating presentable names. ( #9165 )
...
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
57feeab364
Don't ignore set_tweak actions with no explicit value. ( #7766 )
...
* Fix spec compliance; tweaks without values are valid
(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)
* Changelog for 7766.
* Add documentation to `tweaks_for_actions`
May as well tidy up when I'm here.
* Add a test for `tweaks_for_actions`
2020-07-06 11:43:41 +01:00
cc32fa7358
Ensure the body is a string before comparing push rules. ( #7701 )
2020-06-15 16:20:34 -04:00
01294e6b3a
Do not treat display names as globs for push rules. ( #7271 )
2020-04-16 10:52:55 -04:00
Matthew Hodgson