Files
synapse/tests/test_utils/event_injection.py
T
Paul Chobert 8c11b13f63 Fix state events missing from MSC4222 state_after when the since token falls inside a persist batch (#20171)
This PR fixes the issue described as comment here:
https://github.com/element-hq/synapse/issues/18793#issuecomment-3502202379

In Element Call, this shows up as ghost participants: someone who left
the call keeps being displayed until a later state change refreshes the
room.

The bug is not specific to Element Call: any state event can be
affected, RTC membership just changes often enough to make it visible.

## What happens

Alice has a client syncing against a homeserver where events are
persisted by one worker (the event persister) and `/sync` is served by
another (the sync worker). Her client is parked in a long-poll: `GET
/sync?since=s99&timeout=30000`.

Bob joins a call at the same moment Carol sends a message. Carol's
message reaches the persister first; Bob's `m.call.member` arrives while
that write is still in flight, so the per-room persist queue groups them
into one transaction:

```
events (each gets its own stream ordering):
    stream_ordering 100:  m.room.message   Carol
    stream_ordering 101:  m.call.member    Bob        (state)

current_state_delta_stream (how state_after finds state changes):
    stream_id 100 ────►  (m.call.member, @bob) -> $bob_join_call
          ▲
          └─ stamped with the batch MINIMUM (100), not the event's own 101
              (see `_update_current_state_txn`)
```

The transaction commits: both events and the delta row are now in the
database, atomically.

The persister then announces the new events over replication, one RDATA
token per stream ordering — rows are only merged into one token when
they share a position, and 100 and 101 don't. So the sync worker's
events-stream position steps 99 → 100 → 101, and on reaching 100 it
pokes the notifier.

Alice's long-poll wakes at exactly that moment. Her response is built at
the worker's *current* position — `end = 100` — with RDATA 101 still in
the queue:

```
Sync A  (since=99, end=100):
  timeline:     events   99 < ordering ≤ 100  →  [Carol's message]
  state_after:  deltas   99 < stream_id ≤ 100 →  [$bob_join_call]  ← delivered EARLY
  next_batch:   s100                                               ← mid-batch token
```

No race on the client's side is needed: the server *hands out* the
mid-batch token as `next_batch`. Alice's client re-polls with it, as
every sync client does. The worker has meanwhile processed RDATA 101:

```
Sync B  (since=100, end=101):
  timeline:     events   100 < ordering ≤ 101  →  [Bob's m.call.member @101]  ✓
  state_after:  deltas   100 < stream_id ≤ 101 →  []     row is stamped 100   ✗
```

A state event in the timeline with an empty `state_after`. An MSC4222
client trusts `state_after` over timeline state events, so Alice's copy
of Bob's call membership never updates from this response.

On a single process this cannot happen: the batch's stream IDs are
released as a whole, so the position visible to `/sync` jumps 99 → 101
and `s100` is never handed out. Only a process that learns its position
from replication — any sync worker — ticks through the middle of a
batch.


### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2026-09-16 11:07:25 +02:00

208 lines
6.4 KiB
Python

#
# This file is licensed under the Affero General Public License (AGPL) version 3.
#
# Copyright 2020 The Matrix.org Foundation C.I.C
# Copyright (C) 2023 New Vector, Ltd
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# See the GNU Affero General Public License for more details:
# <https://www.gnu.org/licenses/agpl-3.0.html>.
#
# Originally licensed under the Apache License, Version 2.0:
# <http://www.apache.org/licenses/LICENSE-2.0>.
#
# [This file includes modifications made by New Vector Limited]
#
#
from typing import Any, Mapping
import synapse.server
from synapse.api.constants import EventTypes
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
from synapse.events import EventBase
from synapse.events.snapshot import EventContext
"""
Utility functions for poking events into the storage of the server under test.
"""
async def inject_member_event(
hs: synapse.server.HomeServer,
room_id: str,
sender: str,
membership: str,
target: str | None = None,
extra_content: dict | None = None,
**kwargs: Any,
) -> EventBase:
"""Inject a membership event into a room."""
if target is None:
target = sender
content = {"membership": membership}
if extra_content:
content.update(extra_content)
return await inject_event(
hs,
room_id=room_id,
type=EventTypes.Member,
sender=sender,
state_key=target,
content=content,
**kwargs,
)
async def inject_event(
hs: synapse.server.HomeServer,
room_version: str | None = None,
prev_event_ids: list[str] | None = None,
*,
internal_metadata: Mapping[str, Any] | None = None,
**kwargs: Any,
) -> EventBase:
"""Inject a generic event into a room
Args:
hs: the homeserver under test
room_version: the version of the room we're inserting into.
if not specified, will be looked up
prev_event_ids: prev_events for the event. If not specified, will be looked up
internal_metadata: Dict representing the event's internal metadata; see `EventBase.internal_metadata`
kwargs: fields for the event to be created
"""
event, context = await create_event(
hs, room_version, prev_event_ids, internal_metadata=internal_metadata, **kwargs
)
persistence = hs.get_storage_controllers().persistence
assert persistence is not None
await persistence.persist_event(event, context)
return event
async def create_event(
hs: synapse.server.HomeServer,
room_version: str | None = None,
prev_event_ids: list[str] | None = None,
*,
internal_metadata: Mapping[str, Any] | None = None,
**kwargs: Any,
) -> tuple[EventBase, EventContext]:
internal_metadata = internal_metadata or {}
if room_version is None:
room_version = await hs.get_datastores().main.get_room_version_id(
kwargs["room_id"]
)
builder = hs.get_event_builder_factory().for_room_version(
KNOWN_ROOM_VERSIONS[room_version], kwargs
)
(
event,
unpersisted_context,
) = await hs.get_event_creation_handler().create_new_client_event(
builder, prev_event_ids=prev_event_ids
)
# Copy over writable internal_metadata, if set
if internal_metadata:
for key, value in internal_metadata.items():
# Note: this calls the relevant `#[setter]` function in the (Rust) event class'
# internal metadata struct.
# Will reject unknown keys with exceptions.
# This is desirable for our test suite anyway.
setattr(event.internal_metadata, key, value)
context = await unpersisted_context.persist(event)
return event, context
async def persist_message_and_state_event_in_one_batch(
hs: synapse.server.HomeServer,
room_id: str,
sender: str,
) -> tuple[EventBase, EventBase]:
"""Persist a message and a state event in a *single* persist batch (one
`_persist_events_and_state_updates` call), with the message first.
The message's stream ordering is then the batch minimum, so the state
event's `current_state_delta_stream` row (which is stamped with the batch
minimum, see `_update_current_state_txn`) sits *before* the state event's
own position. That is the shape behind
https://github.com/element-hq/synapse/issues/18793.
Both events are created off the same forward extremities, i.e. as
siblings, which is how two events sent concurrently end up in one batch.
Returns:
The persisted (message, state event) pair.
"""
store = hs.get_datastores().main
persistence = hs.get_storage_controllers().persistence
assert persistence is not None
prev_event_ids = await store.get_prev_events_for_room(room_id)
message, message_ctx = await create_event(
hs,
room_id=room_id,
type=EventTypes.Message,
sender=sender,
content={"msgtype": "m.text", "body": "batched message"},
prev_event_ids=prev_event_ids,
)
state_event, state_ctx = await create_event(
hs,
room_id=room_id,
type="m.call.member",
state_key=sender,
sender=sender,
content={"memberships": [{"device_id": "BATCHED"}]},
prev_event_ids=prev_event_ids,
)
await persistence.persist_events([(message, message_ctx), (state_event, state_ctx)])
return message, state_event
async def mark_event_as_partial_state(
hs: synapse.server.HomeServer,
event_id: str,
room_id: str,
) -> None:
"""
(Falsely) mark an event as having partial state.
Naughty, but occasionally useful when checking that partial state doesn't
block something from happening.
If the event already has partial state, this insert will fail (event_id is unique
in this table).
"""
store = hs.get_datastores().main
# Use the store helper to insert into the database so the caches are busted
await store.store_partial_state_room(
room_id=room_id,
servers={hs.hostname},
device_lists_stream_id=0,
joined_via=hs.hostname,
)
# FIXME: Bust the cache
await store.db_pool.simple_insert(
table="partial_state_events",
values={
"room_id": room_id,
"event_id": event_id,
},
)