The Rust federation client now reports Python's
`outgoing-federation-request` span for each request, all its attempts
included, tagged `span.kind`, `peer.address`, `http.method`,
`http.url` and `http.status_code` (for every response, errors
included, as Python sets it before raising). A failed attempt is a log
on it, and a request that fails is marked with the `error` tag. A
request Python's `MatrixFederationHttpClient` sends through the Rust
client is already in Python's span of that name, so Rust makes no
second one (`RequestOptions::traced_by_caller`).
As `inject_header_dict` does, the request carries the span's context
(`uber-trace-id`, and an `uberctx-*` header per baggage item) only to
a server on the opentracing homeserver whitelist, which
`enable_tracing` now takes. A caller that sent Python's context in
`extra_headers` keeps it.
Transactions the native queue sends get the `send_transaction` span
Python's transaction manager makes. It follows from the spans that
queued its EDUs and, as jaeger-client makes the first reference the
parent, is a child of the first, so it shares that trace and its
sampling decision. With no EDU context it starts a trace of its own.
`with_remote_parent` is the layer's way to give a span a parent that
arrived as data. For a whitelisted destination Rust now sets every
EDU's context to `{}`, as `Edu.strip_context` does, after reading
them, where the Python side of the queue stripped them before handing
the EDUs over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
`enable_tracing` now starts a reporter for the homeserver. A finished
span is queued on a bounded channel and dropped if the queue is full,
so reporting never blocks. A `jaeger-reporter` std thread encodes the
spans and sends them as `emitBatch` datagrams to the agent the Python
tracer uses. A batch goes when the next span would not fit in a 65000
byte datagram, at 100 spans, or a second after its first span. Every
batch carries the Python tracer's service name and process tags.
Like jaeger-client, the reporter sends from an unconnected socket and
resolves the agent's host name when it sends: again every 30 seconds
and after a failed send. A name that does not resolve at startup, or an
agent that moves, does not lose Rust spans for good, and an ICMP error
from a down agent cannot fail a later send.
Homeserver shutdown sends the queued spans, waiting up to a second.
The counters are exported as `synapse_rust_tracing_*` metrics, and a
changed send error is logged.
Nothing reports spans yet: the span API comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
Rust work spawned from Python needs the caller's span as the parent of
the spans it will report. `LogContextHandle::capture` now reads the
active span's context (trace ID, span ID, flags, baggage) off the
logcontext's scope, but only once a homeserver has enabled tracing.
Otherwise a capture costs one relaxed atomic load. A capture inside a
task copies the task's handle, so the span stays the one the task was
spawned under.
The `_blocking` entry points ran their futures on the reactor thread
with no handle at all. They now go through
`RustRuntime.block_on_scoped`, which spawns the future with a captured
handle and blocks on the join, the same as `create_deferred`.
`init_tracer` hands the Jaeger tracer's service name, process tags,
agent address and sampler to `RustRuntime.enable_tracing`, which stores
them for the reporter that follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
With `cache_autotuning` configured, `_expire_old_entries` walks Python's
global cache list from the coldest entry and evicts while jemalloc reports
too much memory allocated. Rust caches were not on that list, so they were
never evicted for memory, which matters once the event cache lives in Rust.
Each Rust cache's LRU order is already a sorted run, so the registry
merges them without a global list: `CacheRegistry::evict_coldest` keeps
evicting from the cache whose coldest entry is coldest, up to a bound and
a count. The Python walk calls it before evicting each Python entry, with
that entry's last access as the bound, so both sides go coldest first
together. It reports the coldest Rust entry left, so Python only calls
again once its walk passes that time. Memory is re-read after each batch
of 100 Rust evictions, as after 100 Python ones, and `min_cache_ttl` holds
for Rust entries too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpBQxvqZTUvHCMm3u6CR7J
Invalidation stays in Python. Synapse invalidates caches by *name*: it
resolves `getattr(store, cache_name)` and calls `.invalidate(key)` on
whatever it finds. So `RustCache`, a `#[pyclass]` exposing that protocol,
bound onto a datastore under the right attribute name, inherits the whole
existing path (the replication stream, `txn.call_after`, `invalidate_all`)
with no changes to the machinery. Keys arrive as strings from other workers,
because the stream's column is `keys text[]`, and each key type decodes them
back via `PyCacheKey`. A key it cannot decode matches nothing, as it would
for a Python cache, rather than failing the batch of invalidations it
arrived in.
Every cache belongs to one homeserver's `RustRuntime`, whose `CacheRegistry`
builds it and is what expires it, so nothing here is process-wide. Tests
run several homeservers in one process, and one homeserver's expiry tick
must not reach another's caches. Entries expire by age like Python's, off
the same 30 second looping call and the same `expire_caches` /
`cache_entry_ttl` options, but per cache rather than via a global linked
list, since a cache's LRU order is already its age order. Memory-pressure
eviction is not implemented. That one does need a global coldest-first
order, which a k-way merge over the per-cache LRU tails can give without a
global list.
Rust keeps the hit, miss and eviction counters. `register_rust_cache`
registers a cache with the usual `CacheMetric` and copies them onto it at
each scrape, and wires `set_cache_factor` up so that `SYNAPSE_CACHE_FACTOR_*`
and `caches.per_cache_factors` reach a Rust cache the same way.
There are still no caches. The next commit adds the first.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULdieJyQoHNzeHq42H2M1p
To use `reactor.callFromThread` we need to take the GIL. However, the
vast majority of use-cases on the Rust side involve using that from
Tokio reactor threads, where we do not want to take the GIL (as it can
potentially block for a long time).
In #20252 we added a pure Rust alternative to `callFromThread`.
`run_python_awaitable` is the last usage of `callFromThread`, and so we
replace it with `dispatch_to_twisted`.
---
The diff is a lot smaller if you hide pure whitespace changes.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Tokio tasks currently complete by taking the GIL and calling
`reactor.callFromThread`. Taking the GIL may block for a period of time,
and we do not want that to happen on the tokio reactor threads (as that
can block other work from happening).
To avoid this, we instead add a work queue that we can add to from Rust
without taking the GIL, which is drained by the reactor. We signal to
the Twisted reactor that it should wake up by using a unix socket pair,
which is exactly how `reactor.callFromThread` works.
---
The self-pipe trick is basically where you create a pair of unix sockets
connected to each other. One end is added to the reactor so the reactor
is woken up when there are bytes to read, and when another thread needs
to wake up the reactor it just needs to write a byte into the other unix
socket.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Previously the tokio runtime was stashed in a hidden attribute on the
reactor object, installed lazily by whichever Rust code first needed it,
and started via `callWhenRunning`.
Instead, we create a `RustRuntime` (accessible via
`HomeServer.get_rust_runtime()`) that holds any per-reactor Rust state,
such as the tokio runtime. It is constructed lazily on use. Rust
consumers (`HttpClient`, `VersionsHandler`, the Python DB pool wrapper)
now receive the runtime or reactor handle explicitly, and the
`reactor.run()` / manual-startup workarounds in tests are no longer
needed.
We also add helper wrappers in Rust for `Reactor` and `HomeServer` that
exposes the needed functionality.
The aim is to allow us to have a Rust-side clock (mainly to get the
current time), that respects the unit test per-reactor time management.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
