mirror of
https://github.com/element-hq/synapse.git
synced 2026-09-25 09:04:08 +00:00
This PR fixes the issue described as comment here: https://github.com/element-hq/synapse/issues/18793#issuecomment-3502202379 In Element Call, this shows up as ghost participants: someone who left the call keeps being displayed until a later state change refreshes the room. The bug is not specific to Element Call: any state event can be affected, RTC membership just changes often enough to make it visible. ## What happens Alice has a client syncing against a homeserver where events are persisted by one worker (the event persister) and `/sync` is served by another (the sync worker). Her client is parked in a long-poll: `GET /sync?since=s99&timeout=30000`. Bob joins a call at the same moment Carol sends a message. Carol's message reaches the persister first; Bob's `m.call.member` arrives while that write is still in flight, so the per-room persist queue groups them into one transaction: ``` events (each gets its own stream ordering): stream_ordering 100: m.room.message Carol stream_ordering 101: m.call.member Bob (state) current_state_delta_stream (how state_after finds state changes): stream_id 100 ────► (m.call.member, @bob) -> $bob_join_call ▲ └─ stamped with the batch MINIMUM (100), not the event's own 101 (see `_update_current_state_txn`) ``` The transaction commits: both events and the delta row are now in the database, atomically. The persister then announces the new events over replication, one RDATA token per stream ordering — rows are only merged into one token when they share a position, and 100 and 101 don't. So the sync worker's events-stream position steps 99 → 100 → 101, and on reaching 100 it pokes the notifier. Alice's long-poll wakes at exactly that moment. Her response is built at the worker's *current* position — `end = 100` — with RDATA 101 still in the queue: ``` Sync A (since=99, end=100): timeline: events 99 < ordering ≤ 100 → [Carol's message] state_after: deltas 99 < stream_id ≤ 100 → [$bob_join_call] ← delivered EARLY next_batch: s100 ← mid-batch token ``` No race on the client's side is needed: the server *hands out* the mid-batch token as `next_batch`. Alice's client re-polls with it, as every sync client does. The worker has meanwhile processed RDATA 101: ``` Sync B (since=100, end=101): timeline: events 100 < ordering ≤ 101 → [Bob's m.call.member @101] ✓ state_after: deltas 100 < stream_id ≤ 101 → [] row is stamped 100 ✗ ``` A state event in the timeline with an empty `state_after`. An MSC4222 client trusts `state_after` over timeline state events, so Alice's copy of Bob's call membership never updates from this response. On a single process this cannot happen: the batch's stream IDs are released as a whole, so the position visible to `/sync` jumps 99 → 101 and `s100` is never handed out. Only a process that learns its position from replication — any sync worker — ticks through the middle of a batch. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
208 lines
6.4 KiB
Python
208 lines
6.4 KiB
Python
#
|
|
# This file is licensed under the Affero General Public License (AGPL) version 3.
|
|
#
|
|
# Copyright 2020 The Matrix.org Foundation C.I.C
|
|
# Copyright (C) 2023 New Vector, Ltd
|
|
#
|
|
# This program is free software: you can redistribute it and/or modify
|
|
# it under the terms of the GNU Affero General Public License as
|
|
# published by the Free Software Foundation, either version 3 of the
|
|
# License, or (at your option) any later version.
|
|
#
|
|
# See the GNU Affero General Public License for more details:
|
|
# <https://www.gnu.org/licenses/agpl-3.0.html>.
|
|
#
|
|
# Originally licensed under the Apache License, Version 2.0:
|
|
# <http://www.apache.org/licenses/LICENSE-2.0>.
|
|
#
|
|
# [This file includes modifications made by New Vector Limited]
|
|
#
|
|
#
|
|
from typing import Any, Mapping
|
|
|
|
import synapse.server
|
|
from synapse.api.constants import EventTypes
|
|
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
|
|
from synapse.events import EventBase
|
|
from synapse.events.snapshot import EventContext
|
|
|
|
"""
|
|
Utility functions for poking events into the storage of the server under test.
|
|
"""
|
|
|
|
|
|
async def inject_member_event(
|
|
hs: synapse.server.HomeServer,
|
|
room_id: str,
|
|
sender: str,
|
|
membership: str,
|
|
target: str | None = None,
|
|
extra_content: dict | None = None,
|
|
**kwargs: Any,
|
|
) -> EventBase:
|
|
"""Inject a membership event into a room."""
|
|
if target is None:
|
|
target = sender
|
|
|
|
content = {"membership": membership}
|
|
if extra_content:
|
|
content.update(extra_content)
|
|
|
|
return await inject_event(
|
|
hs,
|
|
room_id=room_id,
|
|
type=EventTypes.Member,
|
|
sender=sender,
|
|
state_key=target,
|
|
content=content,
|
|
**kwargs,
|
|
)
|
|
|
|
|
|
async def inject_event(
|
|
hs: synapse.server.HomeServer,
|
|
room_version: str | None = None,
|
|
prev_event_ids: list[str] | None = None,
|
|
*,
|
|
internal_metadata: Mapping[str, Any] | None = None,
|
|
**kwargs: Any,
|
|
) -> EventBase:
|
|
"""Inject a generic event into a room
|
|
|
|
Args:
|
|
hs: the homeserver under test
|
|
room_version: the version of the room we're inserting into.
|
|
if not specified, will be looked up
|
|
prev_event_ids: prev_events for the event. If not specified, will be looked up
|
|
internal_metadata: Dict representing the event's internal metadata; see `EventBase.internal_metadata`
|
|
kwargs: fields for the event to be created
|
|
"""
|
|
event, context = await create_event(
|
|
hs, room_version, prev_event_ids, internal_metadata=internal_metadata, **kwargs
|
|
)
|
|
|
|
persistence = hs.get_storage_controllers().persistence
|
|
assert persistence is not None
|
|
|
|
await persistence.persist_event(event, context)
|
|
|
|
return event
|
|
|
|
|
|
async def create_event(
|
|
hs: synapse.server.HomeServer,
|
|
room_version: str | None = None,
|
|
prev_event_ids: list[str] | None = None,
|
|
*,
|
|
internal_metadata: Mapping[str, Any] | None = None,
|
|
**kwargs: Any,
|
|
) -> tuple[EventBase, EventContext]:
|
|
internal_metadata = internal_metadata or {}
|
|
if room_version is None:
|
|
room_version = await hs.get_datastores().main.get_room_version_id(
|
|
kwargs["room_id"]
|
|
)
|
|
|
|
builder = hs.get_event_builder_factory().for_room_version(
|
|
KNOWN_ROOM_VERSIONS[room_version], kwargs
|
|
)
|
|
(
|
|
event,
|
|
unpersisted_context,
|
|
) = await hs.get_event_creation_handler().create_new_client_event(
|
|
builder, prev_event_ids=prev_event_ids
|
|
)
|
|
|
|
# Copy over writable internal_metadata, if set
|
|
if internal_metadata:
|
|
for key, value in internal_metadata.items():
|
|
# Note: this calls the relevant `#[setter]` function in the (Rust) event class'
|
|
# internal metadata struct.
|
|
# Will reject unknown keys with exceptions.
|
|
# This is desirable for our test suite anyway.
|
|
setattr(event.internal_metadata, key, value)
|
|
|
|
context = await unpersisted_context.persist(event)
|
|
|
|
return event, context
|
|
|
|
|
|
async def persist_message_and_state_event_in_one_batch(
|
|
hs: synapse.server.HomeServer,
|
|
room_id: str,
|
|
sender: str,
|
|
) -> tuple[EventBase, EventBase]:
|
|
"""Persist a message and a state event in a *single* persist batch (one
|
|
`_persist_events_and_state_updates` call), with the message first.
|
|
|
|
The message's stream ordering is then the batch minimum, so the state
|
|
event's `current_state_delta_stream` row (which is stamped with the batch
|
|
minimum, see `_update_current_state_txn`) sits *before* the state event's
|
|
own position. That is the shape behind
|
|
https://github.com/element-hq/synapse/issues/18793.
|
|
|
|
Both events are created off the same forward extremities, i.e. as
|
|
siblings, which is how two events sent concurrently end up in one batch.
|
|
|
|
Returns:
|
|
The persisted (message, state event) pair.
|
|
"""
|
|
store = hs.get_datastores().main
|
|
persistence = hs.get_storage_controllers().persistence
|
|
assert persistence is not None
|
|
prev_event_ids = await store.get_prev_events_for_room(room_id)
|
|
|
|
message, message_ctx = await create_event(
|
|
hs,
|
|
room_id=room_id,
|
|
type=EventTypes.Message,
|
|
sender=sender,
|
|
content={"msgtype": "m.text", "body": "batched message"},
|
|
prev_event_ids=prev_event_ids,
|
|
)
|
|
state_event, state_ctx = await create_event(
|
|
hs,
|
|
room_id=room_id,
|
|
type="m.call.member",
|
|
state_key=sender,
|
|
sender=sender,
|
|
content={"memberships": [{"device_id": "BATCHED"}]},
|
|
prev_event_ids=prev_event_ids,
|
|
)
|
|
|
|
await persistence.persist_events([(message, message_ctx), (state_event, state_ctx)])
|
|
return message, state_event
|
|
|
|
|
|
async def mark_event_as_partial_state(
|
|
hs: synapse.server.HomeServer,
|
|
event_id: str,
|
|
room_id: str,
|
|
) -> None:
|
|
"""
|
|
(Falsely) mark an event as having partial state.
|
|
|
|
Naughty, but occasionally useful when checking that partial state doesn't
|
|
block something from happening.
|
|
|
|
If the event already has partial state, this insert will fail (event_id is unique
|
|
in this table).
|
|
"""
|
|
store = hs.get_datastores().main
|
|
# Use the store helper to insert into the database so the caches are busted
|
|
await store.store_partial_state_room(
|
|
room_id=room_id,
|
|
servers={hs.hostname},
|
|
device_lists_stream_id=0,
|
|
joined_via=hs.hostname,
|
|
)
|
|
|
|
# FIXME: Bust the cache
|
|
await store.db_pool.simple_insert(
|
|
table="partial_state_events",
|
|
values={
|
|
"room_id": room_id,
|
|
"event_id": event_id,
|
|
},
|
|
)
|