Erik Johnston
26e46786b3
Ports the logcontext classes to Rust, and gives tokio tasks a captured
logcontext so that work running in (or spawned from) Rust is attributed
to the request that caused it.
1. **Add characterization tests for logcontext error messages and the
filter** — pins the exact `logcontext_error` message shapes, the
abuse-detection code paths and `LoggingContextFilter`'s observable
behaviour, *against the existing Python implementation* (this commit is
green on its own). These are the behavioural contract the port has to
satisfy.
2. **Port `ContextResourceUsage` to a Rust pyclass** — self-contained:
the new `synapse_rust.logcontext` module, the class, its stub and the
re-export.
3. **Move the logcontext storage and `LoggingContext` to Rust** — the
core change; the commit message carries detailed design notes.
Highlights:
- The slot is typed `Option<Py<LoggingContext>>`, with `None`
representing the sentinel. `_Sentinel`/`SENTINEL_CONTEXT` stay pure
Python (unchanged); thin wrappers on
`current_context`/`set_current_context` convert at the boundary, and
pyo3's extraction enforces the type (`TypeError` otherwise).
- The accounting is native: one `getrusage(RUSAGE_THREAD)` read per
switch via libc, inline `stop`/`start` bookkeeping for base
`LoggingContext`s, Python dispatch only for subclasses
(`BackgroundProcessLoggingContext`) so their overrides run. The thread
id comes from `PyThread_get_thread_ident` (the exact
`threading.get_ident()`
value) without calling into Python.
- The hot paths avoid per-operation allocation: names are `Py<PyString>`
(the per-log-record `str(context)`/`server_name` reads are INCREF-only),
error branches materialise strings only when hit.
4. **Attribute Rust-spawned work to the caller's logcontext** —
`create_deferred` captures the caller's context and scopes it onto the
spawned task via a tokio task-local (`LogContextHandle`);
`current_context()` gives the task-local read precedence, so
`LoggingContextFilter`/`pyo3-log` resolve the right context on worker
threads with no per-record stamping. `run_python_awaitable` restores the
captured context (via a `with_logcontext` helper, the Rust
`PreserveLoggingContext`) around Python called back from Rust, so e.g.
`runInteraction` from the Rust `/versions` handler accounts its DB usage
against the right request. Integration tests exercise both guarantees
through real production code paths.
Follow-up work on top of this (separate PR): porting
`BackgroundProcessLoggingContext` natively and removing further `Py<_>`
indirections. The fact that `BackgroundProcessLoggingContext` is a
subclass is what forces some of the warts in this PR: e.g. having to use
`Py<LoggingContext>` everywhere, etc.
We don't try (yet) to make this pure Rust, instead we see this as simply
maintaining the Python logcontext machinery when crossing, rather than
trying to make a Rust equivalent that can be used by pure Rust
dependencies. We probably do want to do that in future, as well as wire
up e.g. CPU recording on Rust side, but that is unnecessary for now.
Synapse Documentation
The documentation is currently hosted here. Please update any links to point to the new website instead.
About
This directory currently holds a series of markdown files documenting how to install, use and develop Synapse. The documentation is readable directly from this repository, but it is recommended to instead browse through the website for easier discoverability.
Adding to the documentation
Most of the documentation currently exists as top-level files, as when organising them into
a structured website, these files were kept in place so that existing links would not break.
The rest of the documentation is stored in folders, such as setup, usage, and development
etc. All new documentation files should be placed in structured folders. For example:
To create a new user-facing documentation page about a new Single Sign-On protocol named "MyCoolProtocol", one should create a new file with a relevant name, such as "my_cool_protocol.md". This file might fit into the documentation structure at:
- Usage
- Configuration
- User Authentication
- Single Sign-On
- My Cool Protocol
- Single Sign-On
- User Authentication
- Configuration
Given that, one would place the new file under
usage/configuration/user_authentication/single_sign_on/my_cool_protocol.md.
Note that the structure of the documentation (and thus the left sidebar on the website) is determined by the list in SUMMARY.md. The final thing to do when adding a new page is to add a new line linking to the new documentation file:
- [My Cool Protocol](usage/configuration/user_authentication/single_sign_on/my_cool_protocol.md)
Building the documentation
The documentation is built with mdbook, and the outline of the documentation is determined by the structure of SUMMARY.md.
Building the documentation requires running some python. Ensure you have set up poetry, then run:
poetry install --only docs --no-root
to install just the python dependencies needed to build the documentation.
Then, install mdbook, which will convert the source markdown to HTML pages.
Finally, from the root of the repository, build the documentation with:
poetry run mdbook build
The rendered contents will be outputted to a new book/ directory at the root of the repository. Please note that
index.html is not built by default, it is created by copying over the file welcome_and_overview.html to index.html
during deployment. Thus, when running mdbook serve locally the book will initially show a 404 in place of the index
due to the above. Do not be alarmed!
You can also have mdbook host the docs on a local webserver with hot-reload functionality via:
mdbook serve
The URL at which the docs can be viewed at will be logged.
Synapse configuration documentation
The Configuration Manual page is generated from a YAML file, schema/synapse-config.schema.yaml. To add new options or modify existing ones, first edit that file, then run scripts-dev/gen_config_documentation.py to generate an updated Configuration Manual markdown file.
Build the book as described above to preview it in a web browser.
Configuration and theming
The look and behaviour of the website is configured by the book.toml file at the root of the repository. See mdbook's documentation on configuration for available options.
The site can be themed and additionally extended with extra UI and features. See website_files/README.md for details.