ac26cfac83
Let me give you a clear summary of where this project stands:
...
---
Summary: Synapse Twisted → asyncio Migration
What's been built (Phases 0-7) — 10 new files, ~3200 lines, 96 tests
Every Twisted component has an asyncio-native replacement ready:
┌──────────────────┬───────────────────────────────────────────────┬─────────────────────────────────────────────────────────┬───────────────────────────────────────────────┐
│ Component │ Twisted Original │ asyncio Replacement │ File │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ LoggingContext │ threading.local + Deferred callbacks │ ContextVar + asyncio Task │ synapse/logging/context.py (native functions) │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Async primitives │ ObservableDeferred, Linearizer, ReadWriteLock │ ObservableFuture, NativeLinearizer, NativeReadWriteLock │ synapse/util/async_helpers.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Cache │ DeferredCache │ FutureCache │ synapse/util/caches/future_cache.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Clock │ Clock (Twisted reactor) │ NativeClock (asyncio) │ synapse/util/clock.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Database │ adbapi.ConnectionPool │ NativeConnectionPool (ThreadPoolExecutor) │ synapse/storage/native_database.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ HTTP Client │ treq + Twisted Agent │ aiohttp.ClientSession │ synapse/http/native_client.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ HTTP Server │ JsonResource + Twisted Site │ NativeJsonResource + aiohttp.web │ synapse/http/native_server.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Replication │ LineOnlyReceiver (Twisted Protocol) │ asyncio.StreamReader/Writer │ synapse/replication/tcp/native_protocol.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Event loop │ ISynapseReactor │ ISynapseEventLoop │ synapse/types/__init__.py │
├──────────────────┼───────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ Test helper │ — │ FakeAsyncioLoop │ tests/async_helpers.py │
└──────────────────┴───────────────────────────────────────────────┴─────────────────────────────────────────────────────────┴───────────────────────────────────────────────┘
What's been wired in safely — 224 files changed, 0 regressions
- MemoryReactor type hint → Any across 198 test files (cosmetic)
- synapse/http/server.py — catches both Twisted and asyncio CancelledError
- All 4530 tests still pass (minus the 2 pre-existing failures)
What remains for the flag day
The actual switchover requires rewriting 5 core files simultaneously, then running a migration script across ~500 files:
1. tests/unittest.py + tests/server.py — switch from twisted.trial.TestCase to unittest.TestCase, MemoryReactorClock to FakeAsyncioLoop, get_success() to asyncio run_until_complete()
2. synapse/logging/context.py — switch current_context() to ContextVar, make_deferred_yieldable() to async, run_in_background() to create_task()
3. synapse/util/async_helpers.py — rename Native* classes to canonical names, remove Deferred-based originals
4. Migration script — update all CancelledError, defer.*, Deferred imports across ~500 files
5. pyproject.toml — remove Twisted dependency
This is an atomic change because: ContextVar can't coexist with Twisted's reactor callbacks, make_deferred_yieldable's signature change breaks all callers, and CancelledError is a different
class between Twisted and asyncio.
2026-03-21 16:17:04 +00:00
1bddd25a85
Port Clock functions to use Duration class ( #19229 )
...
This changes the arguments in clock functions to be `Duration` and
converts call sites and constants into `Duration`. There are still some
more functions around that should be converted (e.g.
`timeout_deferred`), but we leave that to another PR.
We also changes `.as_secs()` to return a float, as the rounding broke
things subtly. The only reason to keep it (its the same as
`timedelta.total_seconds()`) is for symmetry with `as_millis()`.
Follows on from https://github.com/element-hq/synapse/pull/19223
2025-12-01 13:55:06 +00:00
5a9ca1e3d9
Introduce Clock.call_when_running(...) to include logcontext by default ( #18944 )
...
Introduce `Clock.call_when_running(...)` to wrap startup code in a
logcontext, ensuring we can identify which server generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.call_when_running(...)` over
`reactor.callWhenRunning(...)`
Part of https://github.com/element-hq/synapse/issues/18905
2025-09-22 10:27:59 -05:00
a31d53b28f
Use twisted.internet.testing module in tests instead of deprecated twisted.test.proto_helpers. ( #18728 )
...
Follows: #18727
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org >
2025-07-30 12:32:10 +01:00
23740eaa3d
Correctly mention previous copyright ( #16820 )
...
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
8e1e62c9e0
Update license headers
2023-11-21 15:29:58 -05:00
2281427175
Use literals in place of HTTPStatus constants in tests ( #13488 )
...
* Use literals in place of `HTTPStatus` constants in tests
* newsfile
* code style
* code style
2022-08-10 11:01:12 -07:00
1595052b26
Use literals in place of HTTPStatus constants in tests ( #13479 )
...
Replace
- `HTTPStatus.NOT_FOUND`
- `HTTPStatus.FORBIDDEN`
- `HTTPStatus.UNAUTHORIZED`
- `HTTPStatus.CONFLICT`
- `HTTPStatus.CREATED`
Signed-off-by: Dirk Klimpel <dirk@klimpel.org >
2022-08-09 14:56:43 +01:00
c97042f7ee
Use literals in place of HTTPStatus constants in tests ( #13469 )
2022-08-08 13:21:27 -07:00
ef3619e61d
Add config settings for background update parameters ( #11980 )
2022-03-11 10:46:45 -08:00
26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size ( #12157 )
2022-03-07 09:44:33 -08:00
e24ff8ebe3
Remove HomeServer.get_datastore() ( #12031 )
...
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
2022-02-23 11:04:02 +00:00
8428ef66c7
Add type hints to synapse/tests/rest/admin ( #11590 )
2021-12-16 14:59:56 -05:00
e5f426cd54
Add type hints to synapse/tests/rest/admin ( #11501 )
2021-12-03 13:57:13 +00:00
35b1900f00
Convert status codes to HTTPStatus in tests.rest.admin ( #11455 )
2021-11-30 09:53:54 +00:00
d08ef6f155
Make background updates controllable via a plugin ( #11306 )
...
Co-authored-by: Brendan Abolivier <babolivier@matrix.org >
2021-11-29 17:57:06 +01:00
7f9841bdec
Lower minumum batch size to 1 for background updates ( #11422 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com >
2021-11-24 19:21:44 +00:00
ea20937084
Add an admin API to run background jobs. ( #11352 )
...
Instead of having admins poke into the database directly.
Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
4ee71b9637
Add some background update admin APIs ( #11263 )
...
Fixes #11259
2021-11-08 16:08:02 +00:00
Matthew Hodgson