`persist_bench.py --opentracing` runs each level with tracing off,
`python` (on, nothing sampled, tracing in Rust patched off),
`unsampled` (on, nothing sampled) or `sampled` (everything sampled),
with spans going to a UDP port nothing reads. `python` against
`unsampled` is what tracing in Rust costs when a request is not
traced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
The persister's entry points and stages now have spans, so the `db.*`
spans of the earlier commits nest under the step that made them:
- `persist.prepare_chunk` and `persist.write_chunk` (with `room_id`
and `events`), created in the functions both the chunk entry points
and the batch call, so they have them at every
`rust_event_persistence` level; `persist.batch_next`,
`persist.batch_abort` and `persist.update_current_state`;
- inside a prepare: `persist.state_delta` (which includes the
extremities), `persist.state_res` (with the resolution's total and
database time), `persist.chain_cover`, `persist.sliding_sync` and
`persist.sticky`;
- inside the write transaction: `persist.txn.read`,
`persist.txn.store_events`, `persist.txn.auth_chain`,
`persist.txn.metadata`, `persist.txn.current_state` and
`persist.txn.sliding_sync`. Most steps only queue their writes, so
their spans time the step's own work, and the writes are timed in
the next round trip's `db.query`.
The spans are created inside the task, so their parent is the Python
span active when the persist was started. `TxnTimings` stay as they
are: Python still needs them for its metrics, which are not sampled.
A trial test makes the test tracer the global one, as `init_tracer`
does, persists a message through the Rust persister and checks what
reaches the agent's UDP port: `persist.batch_next` under the persist
queue's Python span and in its trace, the chunk's steps under it, and
`persist.write_chunk` → `db.persist_events` → `db.connection` →
`db.txn`, with the transaction's steps and `db.query` round trips
carrying queued writes and the COMMIT under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
The read paths now report the same spans as the write path:
- `PoolQuerier` (event fetch, stream ID allocation): `db.connection`
from checkout until it drops, and a `db.query` per statement, tagged
`db.prepared` when the statement missed the connection's cache.
- The sync store's borrowed connections: `db.connection` from checkout
until the drop, and for a transaction a `db.txn` (`db.txn_desc` of
`sync` or `sync_write`, `db.txn_id`, `db.readonly`) with a `commit`
log, or a `db.rollback` child when a dropped transaction rolls back.
Each statement is a `db.query`, with `db.begin` when the pending
BEGIN went with it.
- The state resolution store: `db.connection` → `db.txn`
(`state_res`, read only) → a `db.query` each for the BEGIN, every
read and the COMMIT.
As on the write path and in Python, `db.connection` starts once the
connection is in hand, and the wait for it is a `checked out` log on
the caller's span. A failed `db.query` is marked with the `error` tag
and log. `db.query` spans take the held connection or transaction span
as an explicit parent.
The tests collect spans on a current-thread runtime: the registry
releases a span's enter through the thread's default subscriber, so a
span entered on a worker thread would never close under a test's
thread-local subscriber. Production installs the layer globally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
A write interaction on the native driver now reports spans with the
names and tags of Python's `DatabasePool` spans, so Jaeger searches
and per-operation views work across both:
- `db.{name}` around the whole interaction, every attempt and the
waits for a connection included, with a `checked out` log carrying
each wait and a `retry` log per retry;
- `db.connection` per attempt, from having the connection to giving it
back;
- `db.txn` per attempt, tagged `db.txn_desc`, `db.txn_id` (one per
interaction, as Python names it) and `db.attempt`, with a `commit`
log, and a `db.rollback` child when it rolls back;
- `db.query` per round trip, tagged `db.type` and `db.statement`.
Rust retries on a fresh connection, so unlike Python each attempt has
its own `db.connection`.
One `db.query` covers a round trip rather than a statement, since the
round trip is what the pipelining saves. `db.queued` counts the queued
writes sent ahead of the statement and `db.queued_statements` lists
them, `db.begin` says whether the BEGIN went with it, and
`db.prepared` counts statements prepared first, in a `db.prepare`
child for that extra round trip. Statements are only formatted for
sampled spans.
A span an error leaves is marked as opentracing marks one an
exception leaves: an `error = true` tag and an `error` log.
`_load_next_mult_id`, which runs outside `run_write_interaction`, gets
its `db.{name}` span too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
Rust code now creates spans with the `tracing` crate.
`JaegerLayer` gives each span Jaeger IDs and reports sampled ones
through the homeserver's UDP reporter when they close.
A span's parent is its `tracing` parent, else (for a span with no
explicit parent) the Python span that was active when its task was
spawned, from the task's logcontext handle, else the Python span set
for the thread by `with_py_parent`. With no parent, the configured
`const` or `probabilistic` sampler decides, and a sampled root gets
jaeger-client's `sampler.type` and `sampler.param` tags. A child
copies its parent's trace ID, flags and baggage, so sampling is
decided once per trace and `force_tracing` carries over.
A span ends when it is last exited, or when it closes if it was never
entered. The registry keeps a parent open until its children close, so
its duration must not be taken at close. Work that outlives its parent
should start a trace with `parent: None` and `follows_from`.
Span fields become tags (strings cut to 1024 bytes, as jaeger-client
does), `otel.name` sets the operation name, and events inside a span
become logs. Only Synapse's own spans and events are recorded, not
those of hyper, h2 and other dependencies.
The layer is the process-wide subscriber. It reports to the first
homeserver that enables tracing until that homeserver shuts down or is
dropped. Spans are disabled while no reporter is attached, so they
cost almost nothing with tracing off.
`is_sampled`, `span_context`, `inject_headers` (`uber-trace-id` plus
`uberctx-*` baggage, as jaeger-client's HTTP codec writes them) and
`follows_from` are there for the instrumentation that follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
`enable_tracing` now starts a reporter for the homeserver. A finished
span is queued on a bounded channel and dropped if the queue is full,
so reporting never blocks. A `jaeger-reporter` std thread encodes the
spans and sends them as `emitBatch` datagrams to the agent the Python
tracer uses. A batch goes when the next span would not fit in a 65000
byte datagram, at 100 spans, or a second after its first span. Every
batch carries the Python tracer's service name and process tags.
Like jaeger-client, the reporter sends from an unconnected socket and
resolves the agent's host name when it sends: again every 30 seconds
and after a failed send. A name that does not resolve at startup, or an
agent that moves, does not lose Rust spans for good, and an ICMP error
from a down agent cannot fail a later send.
Homeserver shutdown sends the queued spans, waiting up to a second.
The counters are exported as `synapse_rust_tracing_*` metrics, and a
changed send error is logged.
Nothing reports spans yet: the span API comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
Rust spans will go to the same Jaeger agent as Python's, over UDP, as
the `Agent.emitBatch` oneway message in the thrift compact protocol.
This adds a write-only encoder for that message and the six structs in
`jaeger.thrift`, with no new dependency.
Spans are encoded one at a time so a reporter knows each span's size
before it joins them into a batch, and `emit_batch_len` bounds a
batch's size without building it.
The output matches jaeger-client's own reporter byte for byte. The
golden fixture is generated by
`scripts-dev/generate_jaeger_thrift_fixture.py` with jaeger-client's
generated thrift code and covers every tag type, logs, a follows-from
reference, a 128-bit trace ID, IDs at or above 2^63 (which thrift
sends negative) and long list headers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
Rust work spawned from Python needs the caller's span as the parent of
the spans it will report. `LogContextHandle::capture` now reads the
active span's context (trace ID, span ID, flags, baggage) off the
logcontext's scope, but only once a homeserver has enabled tracing.
Otherwise a capture costs one relaxed atomic load. A capture inside a
task copies the task's handle, so the span stays the one the task was
spawned under.
The `_blocking` entry points ran their futures on the reactor thread
with no handle at all. They now go through
`RustRuntime.block_on_scoped`, which spawns the future with a captured
handle and blocks on the join, the same as `create_deferred`.
`init_tracer` hands the Jaeger tracer's service name, process tags,
agent address and sampler to `RustRuntime.enable_tracing`, which stores
them for the reporter that follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4zxygfJC93SsugxskhoJG
A worker sending a new room's events to an event persister got the
persister's RoomIdCollisionError back as a plain 500. So createRoom did
not retry under a fresh ID, and did not move an alias it had created
off the room that owned the ID. The error now has the M_ROOM_IN_USE
errcode, and the sender raises RoomIdCollisionError again when it sees
it, as it does PartialStateConflictError for a 409.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
A write transaction that failed after prefetching cache stream IDs
finished all of them, but left the unused ones spare. The retry handed
those out first, so it wrote cache invalidations at IDs below the
stream's position, which other workers never fetch. Finishing the
pending IDs now drops the spare ones too.
The prefetch also began its fetch only once the `nextval` rows were
back, so the position could pass an ID still on its way. It now begins
before the reads are sent, as `next_ids` does, and ends unallocated if
they fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Before the write: the sliding sync step's three reads of the room go
out together on one connection, and the relevant state is read once,
not twice; the chain cover step pairs its two input reads, and the
chains' sequence numbers with their links, which are read for the
chains known before allocation since new chains have none; the read of
rejected and soft failed prev events leaves out the chunk's accepted
events, which it could not have found; and a room with no forward
extremities has no current state to read.
After it: get_current_state_deltas groups and budgets the deltas in one
statement, with window functions, so each handler's read is one round
trip in autocommit, and the stream_positions upsert runs in autocommit
too, from both persist paths. The twin test's fake background runner
passes keyword arguments on.
The write transaction read six times between its writes, each read a
round trip: the room's depth, which events it already had, their
redactions, whether the prev events were persisted, the cache prefill's
rejections and redactions, and a cache stream ID. Now the rooms row of a
new room and the depth update are queued writes (the depth update
conditional, as Python's read made it), the three reads it still needs
go out together with eight cache stream IDs in one round trip behind
them, through the new WriteTxn::query_many, and the rest is known: the
cache prefill from the contexts and the redactions already read, and a
prev event the chunk itself persists needs no check. A chunk that
creates a room is written in two round trips instead of eight.
The createRoom bench gets a sane event cache, as the test default of one
entry made every read after persisting a fetch, which no server does.
The rooms row for a new room is inserted by the persist transaction,
Rust or Python, when the create event's context is flagged, instead of
by a store_room transaction of its own beforehand. The insert also
claims the room ID: a row already there fails the batch with
RoomIdCollisionError, and create_room runs the whole creation again
under a fresh ID, perturbing the create event's timestamp for v12+
rooms as the old loop did. The room version is prefilled on the upgrade
path too, as the row is no longer there to read.
The chain cover check skips its rooms read when the create event is
among the events, since the row may not be stored yet, and the
createRoom bench advances the fake clock per room so that rooms do not
collide.
The push rule evaluator wrote each event's push actions to the staging
table, a round trip, for the persister to move into `event_push_actions`
in its transaction. It now leaves them on the event's context, as
`push_actions`, and the Rust persister writes them into
`event_push_actions` in the write transaction, queued behind the other
writes. The staging table stays for the persisters that read from it:
`stage_push_actions` writes the rows there before the Python persister
runs, and before a batch is sent to another worker.
The persist queue's read lock against purges was dropped with a round
trip the request waited for. Its row is now deleted in the background.
Waiters for the write lock retry on their own, so they lose nothing but
a few milliseconds.
The proxy's round trips per room for a room with an invite fall from 81
to 80, and the Python interactions from 14 to 13. The purge lock's two
round trips remain, one of them off the request now; so does
`store_room`, whose insert is also the room ID collision check.
Creating a room read the database from Python for things the request
already knows, each a round trip of its own:
- The creator's and each invitee's profile, for the membership events.
`get_profileinfo` is now cached by user ID, invalidated where the
profile is written: setting the displayname or avatar, registering
with a displayname, and deleting the profile.
- Whether a joining user is already joined, in
`persist_and_notify_client_events`, and whether a membership event's
target is joined, in the push rule evaluator. A membership event that
replaces no membership in the room state, which `replaces_state` in its
unsigned data records, is of a user who is not joined, so there is
nothing to read.
- The new room's version, when the invite is built in Python. The room
handler prefills `get_room_version_id` once the room is stored.
On the native driver with the Rust persister the bench's Python
interactions per room fall from 13 to 11 for a plain room and from 19 to
14 for a room with an invite, and the Postgres round trips the proxy
counts from 72 to 70 and from 86 to 81. `SYNAPSE_BENCH_TRACE` prints a
stack for each interaction with a given description.
A transaction that is one statement then COMMIT takes two round trips.
In autocommit it takes one. On the room creation path these are
`execute`, `simple_insert`, a one-row `simple_insert_many`, the room
version read, the room profile reads and the current state reads.
Multi-row inserts and the current state read of more than 500 keys
run several statements, so they stay in a transaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`store_state_group_chain` inserted the new groups, their edges and their
state one statement at a time, then committed: five round trips for a
new room, with the ID allocation. The inserts, and the bump of the prev
group's pending deletion, are now queued, so a new room's chain takes
two: the IDs, then the rows with the COMMIT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
A join without profile data read the user's display name and then their
avatar URL, two transactions. For a local user with neither in the
content, `get_profileinfo` now reads both at once. A remote user, or
content that has one of the two, takes the same path as before.
If the user has no profile row, the first read used to fail with a 404,
logged, and leave both out. Now both are left out without the log line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
The background update that marks unreferenced state groups for deletion
reads a batch of groups joined to their next groups, then decides each
row in turn: a group is a candidate if every next group is. That needs a
group's next groups decided before it, but the query had no outer ORDER
BY, so the order was whatever the join produced. When a group came
first, its next group looked undecided, the group was kept, and once the
next group was deleted it was left behind for good.
`test_clear_unreferenced_state_groups` caught this about one run in ten
on Postgres, and more often as other changes moved the timing. A next
group is always created after its prev group, so has a higher ID:
ordering by ID descending decides it first. The test now passes every
time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Preparing a chunk worked out the state, then the chain cover index, and
the batch then marked its state groups as persisting: each one waited
for the one before, though none uses another's result.
The chain cover index, a transaction of its own of about six round
trips, is now worked out at the same time as the state and the sliding
sync changes, on its own connection. Marking the state groups runs at
the same time as preparing. A state error or a fallback to Python is
reported as before, ahead of anything the other step found. If the
chunk is handed back to Python, a mark that went ahead is released as a
failure: that only pushes back any pending deletion of the groups, as
Python's own failure path would, and a failure to release is logged
rather than failing the hand-back. The chain cover index is worked out
even when the state step then falls back, which costs chain IDs and can
leave a row in `event_auth_chain_to_calculate` that Python's persist
of the chunk writes and clears again.
The E2EE monitor's room on jki.re now waits on about 44 round trips in
turn, from about 60.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
The sync store's transactions (`SyncReader::begin` and `begin_write`),
which the Rust sync and sliding sync handlers and the persister's reader
use, still sent `BEGIN` on its own and waited for it.
As for `WriteTxn`, the `BEGIN` now goes out with the first statement:
with its execution when the connection has it cached, otherwise with its
prepare. If the `BEGIN` fails, the connection is discarded. A
transaction that runs nothing sends neither `BEGIN` nor `COMMIT`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Marking state groups as persisting locked them with a read, wrote the
marks, then committed: three round trips. Finishing deleted the marks
and the pending deletions, then committed: three more, at the start of
the next batch.
The writes are now queued, so marking takes two round trips (the
read, then the upsert with the COMMIT) and finishing one (BEGIN, the
deletes and COMMIT together).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
The persist transaction sent its statements one at a time: about forty
round trips for a new room, about twenty for a message.
Every write whose result nothing reads is now queued: the batch inserts
and upserts, the deletes and updates, the current state and membership
batches, the sliding sync rows, threads and sticky events. Queued writes
go out together ahead of the next read, or with the COMMIT, in the order
the code issues them, so the statements run in the same order as before.
What is left are the reads, each a round trip carrying the writes before
it: the room's share lock, room depth, outliers, redactions, backward
extremities and the cache stream's IDs.
The partial state insert stays immediate: it turns a constraint
violation into `PartialStateConflict`, so the queue is sent first, and a
queued write's own violation cannot be taken for it. `upsert_one`, for
out-of-band memberships, also stays immediate.
`OwnedColumn::copy` and `owned_params!` copy borrowed values into ones
a queued statement can own (`ToOwnedSql`), and `execute_batch_later`
queues a statement per row.
The new room's write transaction goes from 42 round trips to about 12.
The E2EE monitor's room on jki.re goes from 102 to 79 round trips
overall, a plain room from 90 to 63.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
A `WriteTxn` sent each statement and waited for it before the next, so
a transaction of forty writes took forty round trips, though nothing
read most of their results.
Writes can now be queued with `execute_later`, `insert_many_later` and
`upsert_many_later`, which take owned parameters. Every statement that is
not queued, and the `COMMIT`, first sends the queued writes, in order,
in the same round trip: tokio-postgres sends a request when its future is
first polled, and `join_in_order` polls them in order. So no read can
overtake a queued write, and a transaction ending in queued writes sends
them with its `COMMIT`. Statements not yet prepared on the connection
are prepared together first.
tokio-postgres encodes a request's parameters on its first poll and sends
nothing if that fails, while later requests still go out. So the queued
writes' parameters are checked against their prepared statements' types
before any is sent: otherwise the writes after a bad one, and the COMMIT,
would go through. A dry run sends its queue before rolling back, so it
still checks the writes. The statement methods must not run
concurrently, or a statement could overtake writes queued before it.
A queued write that fails surfaces at the next statement, reported as
its own failure (with an optional mapping, for callers that turn a
constraint violation into a domain error), rather than as the "current
transaction is aborted" error of the statements after it. A failed
attempt drops its queue unsent.
`execute_batch` uses `join_in_order` too: `try_join_all` does not promise
to poll long lists in order.
Nothing queues writes yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Each connection of the native pool caches its prepared statements, and
a statement it has not prepared costs an extra round trip. Two things
made those misses common:
- The pool handed out connections in rotation (deadpool's default FIFO
queue), so each statement was prepared on every connection in turn,
and idle connections were as likely to be picked as busy ones.
- A full cache was emptied whole, so one burst of one-off SQL re-cost
the whole hot set.
The pool now hands out the most recently returned connection first
(LIFO), so the connections that do the work keep warm caches, and a full
cache drops its least recently used statement. The clear counter becomes
`synapse_rust_db_pool_statement_cache_evictions_total`.
With five warm-up rooms, the E2EE monitor's room goes from 135 to 102
round trips.
With LIFO, connections at the back of the pool can sit idle for long
stretches. A server-side close is caught at checkout by the liveness
check, but a firewall that silently drops an idle flow is only noticed
through TCP keepalives, which tokio-postgres enables by default, as
libpq does for psycopg2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Every transaction on the native driver sent `BEGIN` on its own and
waited for it before the first statement: one round trip per
transaction that did nothing but open it.
The `BEGIN` now goes out in front of the first statement's first
request: its execution if the connection has the statement cached, its
prepare otherwise. tokio-postgres queues a request when its future is
first polled, so joining the two puts them on the wire in order, and
Postgres runs the statement inside the transaction. This applies to
Python's transactions (`Connection::with_client`, through a
`PendingBegin` each statement method gets) and to Rust's write
interactions (`WriteTxn`, whose first statement claims the `BEGIN`).
If the `BEGIN` fails, the statement behind it may have run outside a
transaction, so the connection is discarded rather than reused. A retry
could then apply that statement twice, but `BEGIN ISOLATION LEVEL ...`
has no realistic way to fail on the server while the next statement
succeeds: a broken connection fails both. A
`WriteTxn` that runs no statement now sends neither `BEGIN` nor
`COMMIT`.
The E2EE monitor's room on jki.re goes from 150 to 135 round trips, a
plain room from 104 to 93.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`scripts-dev/pg_roundtrip_proxy.py` sits between Synapse and Postgres. It
counts round trips, a client message after a server message on the same
connection, can add latency to each, and records the protocol messages
and SQL of each round trip. SIGUSR2 resets it and SIGUSR1 writes the
counts and log to a file.
`createroom_bench.py --proxy-delay-ms N` runs the server through it and
reports the round trips per request beside the timings. With a delay,
the extra wall time over a run without one shows how many of those round
trips the request waits for in turn. A new `e2ee_invite` scenario is the
room the E2EE monitor creates on jki.re, and `--scenarios` picks a
subset. `persist_bench.Server` honours `PGPORT`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Each connection of the native Postgres pool caches its prepared
statements. A miss costs a round trip to prepare, and the cache is
emptied whole when it reaches 512 statements. Nothing showed how often
either happens, or how often `txn_limit` replaces a connection and loses
its cache.
The pool now counts cache hits, prepares, cache clears, and connections
opened and retired. Python reports them per pool as
`synapse_rust_db_pool_statement_cache_hits_total`,
`..._statement_prepares_total`, `..._statement_cache_clears_total`,
`..._connections_opened_total` and `..._connections_retired_total`,
carried over when the native pool is closed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`createroom_bench.py` takes levels such as `full+rooms`, which add
`rust_room_creation: on` to a `rust_event_persistence` level, and runs
`off`, `full` and `full+rooms` by default. `persist_bench.Server` takes
extra experimental features for that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Behind `experimental_features.rust_room_creation: on`, the room handler
collects the events that follow the create event (the creator's join,
power levels, join rules, history visibility, the initial state, name,
topic) as `(type, state_key, content)`, and `LocalEventSigner` builds,
hashes and signs them all in one call. Each event's prev event, depth
and auth events follow from the one before, as `create_event` gives
them for a batch.
Python keeps every check: the requester checks ran when the create
event was built, and each built event still goes through
`validate_builder`, `validate_new` and the relation check before
joining the batch, which persists through `handle_new_client_event` as
before. Invites are still built by `create_event`.
Rooms the Rust builder does not cover are built in Python and counted
as fallbacks: v1 and v2 rooms, MSC4242 state DAG rooms, appservice
senders, modules with `check_event_allowed`, initial `m.room.aliases`
events, and `m.call.invite` or `m.room.create` in the initial state. If
Rust fails, the events are built in Python. If Python rejects them too,
the request was bad and gets Python's error. Otherwise it counts as a
Rust error. In tests,
`SYNAPSE_TEST_RUST_ROOM_CREATION=on` turns the builder on and makes a
failure fail the test.
Python orders auth events by iterating a set, so the order changes with
the process's hash seed. Rust keeps the order of the auth types. A test
creates rooms of several configurations and room versions in Python,
rebuilds their events in Rust from the same inputs, and compares them
with the auth events taken as a set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Rust could read state groups but not write them. A new
`store_state_group_chain` on the `RustStore` writes the chain
`store_state_deltas_for_batched` stores: an optional full group for the
state before the first event, then one delta group per state event. It
runs in one native transaction, allocating from `state_group_id_seq` as
Python does. A `prev_group` is locked against deletion and its pending
deletion pushed back, and a missing one fails with Python's message.
When the state groups share the main database and that uses the native
driver, `store_state_deltas_for_batched` calls it instead of running a
Python transaction. The context bookkeeping and the cache prefill are
shared by both paths. Room creation, and anything else sending a batch,
saves one Python transaction.
A twin test stores the same batches through both writers and compares
the rows and the groups the contexts get.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Validating a power levels event against its JSON schema took 250 us
even with the validator compiled once, ten times the rest of
`validate_new`.
Rust now answers "certainly valid" or "not sure" for the power levels
schema, the mentions schema and the canonical JSON check. Python runs
its own check only when Rust is not sure, so Python still decides every
rejection and its error message. Rust only accepts integers it parsed as
`i64` in range, so floats such as `50.0`, which the schema accepts, are
left to Python.
`_check_size_limits` takes the canonical JSON length from Rust instead
of building the dict and encoding it. Rust keeps each number's source
text, which can differ from how Python prints a float, so an event with
any number other than a plain integer is still measured in Python.
`validate_new` on a default power levels event: 305 us before, 43 us
after.
Tests generate random contents and check that whatever Rust accepts,
Python accepts too. They found that the strict canonical serializer in
`canonical_json.rs` does not range-check numbers inside arrays, so the
canonical check walks the JSON itself. Nothing else uses strict mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`create_local_event_from_event_dict` encoded the event as canonical JSON
three times: once for the content hash, once to sign the redacted event,
and once to parse it as an `Event`, which then hashed it again for the
event ID.
A Rust `LocalEventSigner`, one per homeserver, now takes the canonical
JSON once, adds the content hash and the signature, and builds the
`Event`. The signed bytes are the redacted event without its signatures,
which is also what the reference hash covers, so the event ID comes from
them without another pass. Hashes and signatures already on the event
(a make_join template) are kept, as before.
Numbers keep the text Python's encoder gave them, so floats and large
integers in old room versions hash and sign as they did. An event the
room version cannot parse is still a 400. A make_join template with a
non-object `hashes` or `signatures` entry, or no `content`, is now a
400 too, where it used to raise a 500.
Saves about 45 us per event on this host. ed25519 signing is 25 us of
what is left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
With `require_membership_for_aliases` (the default), `create_event`
checks that the sender of an `m.room.aliases` event has joined, by
loading their membership event from the database. Since the initial
events of a room are persisted in one batch, the creator's join is not
there yet, so a `/createRoom` with `m.room.aliases` in its initial
state failed with 403 "You must be in the room to create an alias for
it".
Persist the create event and the join on their own first in that case,
as for v1 rooms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`/createRoom` sent each invite with `update_membership_locked` after the
room's initial events, one persist round per invitee.
Invites to local users now go into the batch with the rest of the room, in
`invite` order, until the first invite that has to be sent on its own:
- an invite to a remote user, which needs a federation request first;
- an invite that would be refused (server notices user, non-admin invites
blocked, the invitee's invite rules). It is sent the old way once the room
exists and fails the request as before.
`RoomMemberHandler.build_new_room_invite_content` makes those checks and
builds the content, with the invitee's profile where the server includes
it in invites. A repeated invitee is skipped, as `deduplicate_state_event`
dropped the repeat before.
Each batched invite is also checked against the auth rules before it joins
the batch, as one that fails (inviting yourself, or a power level override
that forbids invites) would otherwise abort the whole batch and leave the
room without events. It and the invites after it are sent on their own
instead, and fail the request once the room exists, as before. The
`trusted_private_chat` preset still gives every invitee ops in rooms
without MSC4289 creators.
Modules that check invites (`user_may_invite`) or events
(`check_event_allowed`) are given the room state from the database, so
with any of those registered all invites are sent on their own. So are
invites in v1 and v2 rooms, whose create event and join are already
persisted on their own.
`invite_room_state` (and `knock_room_state`) now take earlier events of the
same batch from the batch, as they are not in the database yet. Invitees
are passed to the notifier as extra users, as `update_membership` did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
After `batch_persist_unpersisted_contexts`, an event's `EventContext` only
had state group IDs, so every read of the state before the event went to
the state store. For a new room that was eight state-group reads per
`/createRoom`: six from `deduplicate_state_event`, one from the
create-event check in `persist_and_notify_client_events`, and one from the
persister working out the new current state.
Keep the state map the batch was built from on the `EventContext`, as a
private attribute that `get_prev_state_ids` prefers and that is not
serialized, so a context sent to another worker reads the state group as
before. A context that arrives with a state group of its own, which the
batch then replaces, drops its map. `store_state_deltas_for_batched` also prefills the state group
caches with the state after the last event of the batch, through the
helper `store_state_group` now shares.
On the trial harness (SQLite) a plain room drops from 35 to 27 database
interactions, and `/createRoom` from 17 to 10 database transactions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`/createRoom` persisted the create event on its own, then the creator's
join through `update_membership`, then the rest of the initial state as a
batch. Each round paid for the persist queue, the purge lock, forward
extremities, chain cover, a replication notify and a wake-up of the stats
and presence pollers.
Build the create event and the join against the in-memory state map like
the other initial events, and persist all of them in one batch. The join's
content comes from `RoomMemberHandler.build_creator_join_content`, which
shares the suspension and profile handling with `update_membership_locked`
through the new `_prepare_membership_content`. The MSC4291 create event,
built before the room ID is known, now also leaves its context unpersisted
for the batch.
`store_state_deltas_for_batched` accepts a batch that starts the room: it
stores the empty state before the create event as a full state group, then
the delta chain. If the first context already has a group for the state
before it (a module rewrote the event), the chain builds on that instead.
`calculate_context_info` and `UnpersistedEventContext` treat an empty state
map as known empty state. Before, an empty map tripped an assertion. Room
creation never passed one, but federation could (`compute_event_context`
with an empty remote state), and such an event is now stored with an empty
state group and left to the auth checks.
Two cases keep persisting the create event and the join on their own:
v1 and v2 rooms, whose builder reads prev and auth event hashes from the
database, and servers with a `check_event_allowed` module callback, which
is given the room state from the database and may have the event rebuilt.
On the trial harness (SQLite) a plain room drops from 68 to 35 ms and 26
to 17 database transactions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`jsonschema.validate` checks the schema against the metaschema on every
call. For power levels that cost more than validating the event: about
9 ms per `/createRoom`. Check the power levels and mentions schemas once
at import and validate with bound validator instances, reporting the same
best-match error.
`create_event` also ran `validate_new` a second time on the event that
`create_new_client_event` had just validated. Drop the repeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`tests/rest/client/bench_createroom.py` times room creation per room in
the trial harness, with optional cProfile dumps and per-room counts of
database interactions and SQL statements. It skips unless
`SYNAPSE_BENCH_CREATEROOM` is set.
`scripts-dev/createroom_bench.py` times it against a real server, reusing
`persist_bench.Server`. That server now lifts the room-creation rate limit
and can run on psycopg2 instead of the native driver.
The test template database's maintenance connection now always uses
psycopg2, as `setup_test_homeserver` does. The Rust driver's `postgres`
module has no `connect`, so it failed under `SYNAPSE_TEST_RUST_DRIVER=1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`twisted.internet.task.Clock` re-sorts its whole list of pending calls on every
`callLater` and after every call it runs. A test homeserver has dozens of
pending calls and the threadless database pool schedules a `callLater(0, ...)`
per transaction, so this showed up as ~10% of a test's time in profiles.
Insert new calls in order with `bisect.insort` and only re-sort after a call
was rescheduled with `reset` or `delay`.
Together with the previous commit, `tests.rest.client.test_rooms` goes from
44.9s to 32.7s and `tests.handlers` from 146s to 114s single-process on
SQLite; a 147-test Postgres subset from 85.9s to 74.4s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Every test homeserver runs the background updates when it starts, and with a
fresh copy of the template database there are dozens to run: 29ms per test
on SQLite and 99ms on Postgres, against a median test time of ~140ms.
Run them once per process on a throwaway homeserver and copy the result back
over the template (`backup()` for the in-memory SQLite template, `CREATE
DATABASE ... TEMPLATE` for the Postgres base database), so each test's
homeserver finds nothing left to do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Trial worker memory grew steadily over a run because `HomeServer`s outlived
their test. Each one is a ~10k-object reference cycle; GC is disabled while a
test runs and the tearDown only does a gen-0 collection, which frees the
homeserver only if nothing older still points at it. Four things did:
- `ThreadedMemoryReactorClock.__init__` installs `tls._get_default_clock =
lambda: self` into the `twisted.protocols.tls` module. That held the test
reactor, whose never-removed shutdown triggers hold handlers, and so the
homeserver, until the next test's reactor replaced it. Hold the reactor
weakly instead.
- `synapse.config.cache._CACHES` maps cache names to bound `set_cache_factor`
methods, so an old homeserver lived until a later test registered a cache
with the same name. Hold bound methods through `weakref.WeakMethod`.
- `_CacheContext._cache_context_objects` is a class-level `WeakValueDictionary`
keyed on `(cache, key)`: the key strongly holds the cache, which holds the
homeserver, for as long as any context exists. Key on `id(cache)`; the id
cannot be reused while an entry exists since the context holds the cache.
- trial keeps finished `TestCase` instances around (for example in its list of
skipped tests), and those keep `self.hs`, `self.store`, ... Drop every
attribute a test added from setUp onwards before the gen-0 collection.
The fake reactor's `callWhenRunning` and shutdown hooks are emptied once the
homeserver has shut down: they never fire in tests, and the Rust runtime holds
the reactor through a reference the garbage collector cannot see, so a reactor
that still pointed at the handlers kept the homeserver alive.
With this the homeserver dies in the existing gen-0 collection. A full
collection per test was ruled out: it costs ~100ms per test. RSS over
`tests.rest.client.test_rooms` goes from 148→337 MB to 148→176 MB, and over
`tests.handlers` from 170→498 MB to 170→278 MB.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
Brings Rust sliding sync (erikj/rust-sss-queries) together with Rust
event persistence (erikj/rust-persist-validate). Both branch from
a78b02b55c.
The conflicts were both sides adding parallel entries:
- the Complement and test environment switches
- the `RustHandlers` fields and getters
- the `_BlockingRustCalls` types
These keep both. `RustHandlers` builds the persister and the sliding sync
handler in one pass over the native pool. `SyncReader::begin` keeps the
persist side's flag-before-BEGIN and the sliding sync side's transaction
count and `begin_write`. Both sides had added the `invite_room_state`
and `knock_room_state` accessors to `Unsigned`, so one copy is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVLsNh6akt4ifEizY4D1ta
`scripts-dev/persist_bench.py` starts a real Synapse process per
`rust_event_persistence` level, on a fresh Postgres database with the
native driver, and runs the same workloads through the client API:
concurrent messages, membership churn, joins into a room with many
state events, and a redaction storm. For each, it reports wall time,
the CPU of the reactor thread and of the process, the change in
resident memory, and the `persist_events` transactions from the metrics
endpoint.
On this sandbox, 20 users, two runs of each level:
workload level ms/event txn ms proc CPU s
messages off 19.7 56.7 29.1
messages full 14.4 4.2 23.6
membership off 49.6 7.4 9.5
membership full 41.3 4.4 8.0
big_state_joins off 58.1 7.7 1.05
big_state_joins full 50.5 4.7 0.93
redactions off 13.1 38.5 3.2
redactions full 11.5 7.1 3.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The native driver's write path recorded no `sql_txn_count` or
`sql_txn_duration`, nor the log context's transaction counts, so the
`persist_events` panels read near zero at `rust_event_persistence:
transaction` and `full`.
`run_write_interaction` now times each interaction as `new_transaction`
does: every attempt, from when it has its connection, so the wait for
one is left out. The times go into a task-local list that
`with_txn_timings` collects. The persister's entry points carry the list
back on what they already return (`PersistEffects`, `PreparedChunk`,
`PersistedBatch`, `CurrentStateUpdate`), and Python records it where
`new_transaction` would: the two counters, the pool's perf counters and
the current log context, which is the caller's when Python applies what
Rust returned. That costs no GIL hop.
When a persist fails, or Python gives up a batch, no result carries the
times back, so the persister keeps them and `flush_rust_persist_metrics`
counts them at the next scrape, without a log context. In shadow mode
Python's own interactions are the ones used, so Rust's dry run is not
recorded. The stream orderings' `nextval`, which runs outside a write
interaction, is recorded as `_load_next_mult_id`, as Python's is. The
`db.<desc>` tracing span is not recorded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`_update_current_state`, the queue's `_UpdateCurrentStateTask`, becomes
one call to `Persister.update_current_state`:
- `calculate_current_state` reads the room's forward extremities (its
state DAG extremities in an MSC4242 room) and takes the state of
their one state group, or resolves several, after checking the groups
still exist and putting off their deletion, as `resolve_state_groups`
does.
- The delta from the current state and the sliding sync table changes
are computed as for a chunk, with no events.
- If anything changed, one stream ID is taken from the events stream,
as `get_next` does, and `_update_current_state_txn` runs in Rust.
Python applies the effects, which finishes the ID.
A room on state resolution v1 whose extremities need resolving is handed
back, and Python recalculates its state as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
`_after_persisting_events` moves into the batch step, after the write
and before Python applies the chunk and finishes its stream orderings,
as it ran inside `get_next_mult` in Python:
- The persisted events, by type and origin, and the persisted position
go to the Rust metrics that `flush_rust_persist_metrics` reports.
- With MSC4293, a join, or a leave by someone else, ends the redaction
of the target's events. A ban or kick with `redact_events` set, by a
sender who may redact (`_can_sender_redact`, read from the current
state), starts it. Python then drops the room's events from its event
cache, through a new `event_cache_room_invalidations` effect, even if
a later event of the chunk fails, as Python drops them right after
each start.
- The forward extremities prefill moves into the effects.
The MSC4293 flag goes with each batch, as Python reads it per chunk and
tests change it at runtime.
If what follows the write fails, the chunk is still written, so Python
applies its effects and finishes its orderings as a failure, as
`get_next_mult` does when its block raises, then raises the error and
releases the batch's state groups as a failed persist does.
Where Python would raise on odd power levels or memberships, Rust is
more lenient: it reads a level that is a string of an integer as that
number (Python compares two strings, or raises), a null default as 0
and a `users` or `events` that is not an object as missing, and skips a
member event without a membership. Numbers, floats and bools compare as
in Python.
The chunk path, for chunks Rust hands back, keeps the Python block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
At `full`, `_persist_event_batch` hands the batch to
`Persister.persist_batch` and steps through it with `batch_next`:
- The first step drops the events already persisted under the same
transaction ID (`get_already_persisted_events`).
- Each step prepares the next chunk of up to 100 events, marks the
batch's state groups as being persisted
(`persisting_state_group_references`, two short native
transactions), takes the chunk's stream orderings and writes it.
- Python applies what the chunk asks for, runs
`_after_persisting_events` and finishes the orderings before the next
step, which first releases the state groups. So each chunk is
prepared against current caches.
- If Python cannot apply a chunk, `batch_abort` releases the state
groups as a failed persist does.
A chunk Rust cannot prepare (a state resolution fallback, or an error)
is handed back with the rest of the batch to the chunk-by-chunk path,
which prepares it again and falls back or raises as at lower levels.
Python drives the steps, rather than Rust calling back into Python, so
every step has a `_blocking` twin for tests and nothing waits on the
reactor from Rust. That costs the same hops as a callback: one in and
one out per chunk.
The prepare inputs are now built from `Event`s that share their data
with the Python events, so a chunk's input can be built without the
GIL once the batch has been deduplicated.
The persister's Deferreds already follow the logcontext rules, so they
are awaited directly. Wrapping them in `make_deferred_yieldable` again,
as the `prepare_chunk`, `persist_events_txn` and `persist_chunk` call
sites did, resumed the caller in the sentinel context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
At `full`, `Persister.persist_chunk` takes the chunk's stream orderings
from the events or backfill stream tracker, which Python's
`_stream_id_gen` and `_backfill_id_gen` share. It runs `nextval` in
autocommit after prepare, as late as `get_next_mult` does, and sets each
event's stream ordering, received time and instance before the write
transaction. Backfilled events take theirs negated.
The orderings come back unfinished in `PersistEffects.stream_ids`.
`_persist_events_and_state_updates` skips `get_next_mult` when Rust takes
them. Its post-transaction block (metrics, MSC4293
`room_ban_redactions`, the extremities prefill) moves to
`_after_persisting_events`, which both paths call. At `full` it runs
after the effects are applied and before the orderings are finished, as
it ran inside `get_next_mult`. So a ban that redacts a user's events is
recorded before other workers see it.
`finish_rust_persist_stream_ids` finishes the orderings as
`get_next_mult` does on exit: it tells replication, and on success moves
`stream_positions` on. If the transaction fails, Rust finishes them. If
the apply or the follow-ups raise, Python finishes them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
The seventh commit of phase 2, which completes it. The Rust transaction
now covers the whole of `_persist_events_txn`, so at the `transaction`
level every chunk Rust prepared is written in Rust, including on a
server with `include_profile_updates_in_sync`.
- `txn/profile.rs` ports `_record_profile_updates_for_state_delta_txn`
and the helpers it calls: the joined room updates of local members
who joined, and the left room updates, both ways, of members who left
or were state reset out of the room, with the deletes of the rows they
replace. It runs at the end of `update_current_state`, as in Python.
- Its rows take IDs from the profile updates stream inside the
transaction, through a third `TxnStream` over the tracker of
`_profile_updates_id_gen`. `PersistEffects.profile_stream_ids` hands
them to the applier, which finishes them as `get_next_mult_txn` does,
as it does the sticky events stream's. The two share
`_finish_txn_stream_ids`.
- The controller no longer falls back to Python on a server that
records profile updates.
- The Python helpers take the members in user ID order where they took
a set's order, so both sides take the stream IDs in the same order.
- `Persister.persist_events_txn` resolves with the effects alone. The
events left and the redacted copies were only for the Python steps
after Rust's in the twin test, which are gone.
The twin test compares the profile updates rows each twin's persists
wrote, and the rows of each twin's own users from outside them, which a
leave deletes. Those are compared without `inserted_ts`, as each twin
writes them at a different time. New scenarios, with members of each
twin's own, cover:
- joins, a change of display name, and leaves
- several members in one chunk
- a member state reset out of the room
- a remote member state reset out of it, which writes nothing, as only
local members have membership snapshots
- a leave while still sharing another room with some members, and with
all of them
- a leave deleting the rows about earlier profile changes
- the last local member leaving
Making the Rust shared room query match nothing, dropping the reverse
left room rows, or skipping the delete of the leaver's rows about the
others, fails the scenarios that cover it. The
prereqs test now checks that a server recording profile updates writes
the rows in Rust and finishes the stream IDs.
These suites pass at `SYNAPSE_TEST_RUST_PERSIST=transaction` on the
native driver: `tests.handlers` (including the profile and sync tests
that record profile updates), `tests.federation`, `tests.replication`,
`tests.rest.client.sliding_sync` (including the profiles extension),
the phase 2 storage suites, and the `tests.rest.client` group but for
the two `test_rooms` tests that count database transactions, which fail
the same way at `prepare`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXMkcZVkcZhi21augBff4a
