From d355c9bd97b0799204ef26df952a66e13e75026a Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 23 Mar 2026 10:36:29 +0000 Subject: [PATCH] track the migration --- MIGRATION_ASYNCIO.md | 145 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 MIGRATION_ASYNCIO.md diff --git a/MIGRATION_ASYNCIO.md b/MIGRATION_ASYNCIO.md new file mode 100644 index 0000000000..a30f95ee73 --- /dev/null +++ b/MIGRATION_ASYNCIO.md @@ -0,0 +1,145 @@ +Asyncio Migration: Synapse Twisted → asyncio + +Overview + +This document describes the complete migration of Synapse's Matrix homeserver from Twisted to asyncio + aiohttp. The work was done incrementally across multiple sessions, starting from a codebase with ~500 files importing + Twisted. + +Architecture + +Before + +- Event loop: Twisted reactor (with asyncioreactor bridge to asyncio) +- HTTP server: Twisted Resource/Site/Request tree +- HTTP client: Twisted treq + Agent + IResponse +- Database: Twisted adbapi.ConnectionPool +- Caches: DeferredCache using Twisted Deferred +- Async primitives: Deferred, defer.ensureDeferred, defer.gatherResults +- Replication: txredisapi (Twisted Redis client) +- Tests: Twisted trial.unittest, MemoryReactorClock, nest_asyncio + +After + +- Event loop: Native asyncio (asyncio.run(), asyncio.get_event_loop()) +- HTTP server: aiohttp.web with SynapseRequest compatibility shim +- HTTP client: aiohttp.ClientSession (NativeSimpleHttpClient) +- Database: NativeConnectionPool using asyncio.loop.run_in_executor() +- Caches: FutureCache using asyncio.Future +- Async primitives: asyncio.Future, asyncio.ensure_future, asyncio.gather +- Replication: redis.asyncio (redis-py async) +- Tests: unittest.IsolatedAsyncioTestCase, native async test methods + +Key Components Migrated + +1. Database Layer (synapse/storage/) + +- NativeConnectionPool (synapse/storage/native_database.py): asyncio-native DB pool using ThreadPoolExecutor + loop.run_in_executor(). Thread-local connections for normal DBs, shared connection for :memory: SQLite. +Single-worker executor for shared connections to prevent deadlocks. +- DatabasePool (synapse/storage/database.py): Uses NativeConnectionPool instead of adbapi.ConnectionPool. Fresh in-memory DB copy per test via sqlite3.Connection.backup() for test isolation. + +2. Cache Layer (synapse/util/caches/) + +- FutureCache (synapse/util/caches/future_cache.py): asyncio-native replacement for DeferredCache. Uses asyncio.Future and ObservableFuture instead of Deferred/ObservableDeferred. +- ObservableFuture (synapse/util/async_helpers.py): Wraps asyncio.Future so multiple observers can await the same result. Includes callback()/errback() compat methods for code that previously used Deferred API. +- Cache descriptors (synapse/util/caches/descriptors.py): DeferredCacheDescriptor uses asyncio.ensure_future() instead of defer.maybeDeferred. + +3. HTTP Server (synapse/http/) + +- aiohttp_shim.py (NEW): Compatibility shim providing SynapseRequest (wraps aiohttp.web.Request), SynapseSite (data-only config), ShimRequestHeaders/ShimResponseHeaders, aiohttp_handler_factory (catch-all aiohttp +handler), _resolve_resource (resource tree traversal). +- resource.py (NEW): Pure-Python Resource base class replacing twisted.web.resource.Resource. Provides putChild/getChild/children tree structure. +- server.py: Removed Resource inheritance from _AsyncResource/JsonResource. Removed NOT_DONE_YET, _ByteProducer, failure.Failure. Response helpers write directly to request buffer. StaticResource serves files without +Twisted's File. +- site.py: Now a thin re-export layer from aiohttp_shim.py. + +4. HTTP Client (synapse/http/) + +- NativeSimpleHttpClient (synapse/http/native_client.py): aiohttp-based replacement for SimpleHttpClient. Supports IP blocklisting via custom DNS resolver, proxy support, lazy session creation. Aliased as SimpleHttpClient + for drop-in replacement. +- NativeReplicationClient (synapse/http/native_client.py): Routes synapse-replication:// URIs to worker instances via TCP or UNIX sockets. +- Federation client (synapse/http/matrixfederationclient.py): Uses aiohttp session with SRV resolution and well-known delegation instead of Twisted MatrixFederationAgent. + +5. Clock & Async Helpers (synapse/util/) + +- NativeClock (synapse/util/clock.py): asyncio-native clock with fake time support. sleep() uses internal _pending_sleeps heap for deterministic testing. advance() fires pending sleeps. NativeLoopingCall uses asyncio +tasks. Clock is now an alias for NativeClock. +- async_helpers.py: timeout_deferred → asyncio.wait_for, delay_cancellation → asyncio.shield, gather_results → asyncio.gather, Linearizer/ReadWriteLock/AwakenableSleeper → asyncio-native implementations. + +6. Logging Context (synapse/logging/context.py) + +- Uses contextvars.ContextVar for current context tracking. +- run_in_background returns asyncio.Task for coroutines, resolved asyncio.Future for plain values, failed Future for exceptions. +- make_deferred_yieldable is async def — awaits any awaitable with logcontext preservation. +- logcontext_error has re-entrancy guard to prevent infinite loops when logging triggers context switches. + +7. App Startup (synapse/app/) + +- _base.py: listen_http() creates aiohttp.web.Application with the shim handler. start_reactor() uses asyncio event loop. Event loop created early in main() before homeserver setup. +- homeserver.py: Creates asyncio event loop before constructing homeserver. Server listens on aiohttp TCPSite/UnixSite. + +8. Redis/Replication (synapse/replication/tcp/) + +- redis.py: Replaced txredisapi with redis.asyncio. RedisSubscriber uses pubsub.listen() async iterator. RedisReplicationManager handles reconnection with asyncio tasks. +- external_cache.py: Uses redis.asyncio.Redis directly with native await. + +9. Rust Extension (rust/src/) + +- http_client.rs: Replaced twisted.internet.defer.Deferred with asyncio.Future. create_future() uses asyncio.get_event_loop().create_future() and loop.call_soon_threadsafe() to deliver results from Tokio threads. +- http.rs: Functions renamed in docs but kept same signatures — they work with any Python object that has content/uri/method/requestHeaders attributes (compatible with both the aiohttp shim and legacy Twisted). + +10. Test Infrastructure (tests/) + +- TestCase extends unittest.IsolatedAsyncioTestCase — each test gets its own event loop managed by the framework. +- asyncSetUp: Homeserver setup is async — await self.make_homeserver(...), await setup_test_homeserver(...). +- make_request: Now async def — dispatches request and awaits the handler task directly. No more await_result pump loop. +- get_success(d): Now async def — just await d. No more event loop pumping. +- register_user, login: Now async. +- FakeReactor (ThreadedMemoryReactorClock): Pure-asyncio, linked to NativeClock via set_clock(). advance() delegates to clock. +- DB isolation: Each test gets a fresh in-memory SQLite via sqlite3.Connection.backup(). + +Dependencies Removed + +- Twisted: No longer a runtime dependency. All imports wrapped in try/except ImportError. +- treq: Removed entirely. +- txredisapi: Replaced with redis.asyncio (redis-py). +- matrix-synapse-ldap3: Disabled (depends on Twisted). LDAP loaded dynamically. +- nest_asyncio: No longer needed — IsolatedAsyncioTestCase handles async tests natively. + +Dependencies Added + +- aiohttp: HTTP server and client. +- redis (redis-py): Async Redis client for replication. + +Remaining Twisted References (11 files, all guarded) + +All wrapped in try/except ImportError with asyncio fallbacks: +- synapse/http/client.py — legacy utility stubs +- synapse/http/connectproxyclient.py — proxy credential classes +- synapse/http/server.py — StaticResource fallback +- synapse/http/__init__.py — QuieterFileBodyProducer stub +- synapse/app/_base.py — manhole listener +- synapse/app/complement_fork_starter.py — test forking +- synapse/crypto/context_factory.py — TLS factories +- synapse/replication/tcp/protocol.py — TCP replication protocol +- synapse/replication/tcp/redis.py — Redis protocol stubs +- synapse/replication/tcp/resource.py — replication server factory +- synapse/util/manhole.py — SSH debug console + +Key Design Decisions + +1. Compatibility shim over rewrite: The aiohttp_shim.py provides a SynapseRequest that presents the same API as Twisted's Request — 100+ REST endpoint files needed zero changes. +2. Resource tree preservation: The putChild/getChild tree structure was kept via a pure-Python Resource class, with _resolve_resource() for path traversal. +3. Fake time via NativeClock: Tests control time deterministically via clock.advance(). The FakeReactor.advance() delegates to NativeClock.advance() so both share the same time source. +4. IsolatedAsyncioTestCase: Each test gets its own event loop. Test methods are async def and use await directly. No nested run_until_complete, no nest_asyncio, no pump loops. The event loop drives ALL tasks naturally. +5. Single-worker executor for shared connections: In-memory SQLite test DBs use max_workers=1 to prevent concurrent access deadlocks on the shared connection. +6. Cookie handling in shim: request.cookies list is flushed to Set-Cookie response headers in finish(), matching Twisted's automatic behaviour. +7. Form body parsing: The shim's prepare_for_dispatch() merges form-encoded POST bodies into request.args, matching Twisted's automatic behaviour. + +Known Issues / Future Work + +- Session lifetime off-by-one: Token expiry check uses < not <=, so tests advancing exactly to the expiry boundary need +1 second. +- Ratelimit pause: Uses clock.sleep() (fake time) — tests need reactor.advance() to fire the sleep. +- Background process logging context: BackgroundProcessLoggingContext.start() can be called on finished contexts. Re-entrancy guard in logcontext_error prevents infinite loops. +- Production server TLS: Live TLS certificate rotation not supported with aiohttp (was a Twisted reactor feature). Use a reverse proxy. +- Manhole: Still requires Twisted's SSH libraries. Could be replaced with asyncssh. +- Complement test forking: complement_fork_starter.py still references Twisted reactor types for compatibility.