Now that we do a bit more validadtion of events, it's possible that an
event persisted in the database may now not pass validation. This
shouldn't happen, but let's handle it correctly by logging and returning
that we couldn't find the event.
This is the same as what we do if we can't parse the JSON.
Adapt tests that mutated Python event internals (`_event_id`, `_dict`,
direct attribute assignment, `FrozenEventV3(...)` construction) to work
with the new Rust-backed `Event` class:
- Rebuild events via `make_event_from_dict` / `make_test_event` instead
of patching attributes in place.
- Plumb `rejected_reason` through `_join_rules_event` rather than
assigning to `rejected_reason` after construction.
- Replace the hand-built event in `test_msc4242_state_dag` with a
`Mock(spec=EventBase)` since the test only needs a handful of
attributes.
- Add `# type: ignore` for the deprecated `event.user_id` / `event[key]`
accessors and for assigning to `event.content`.
- In `make_test_event`, drop the default `room_id` for v11+ create
events so each gets a distinct hash-derived room ID.
Replace the abstract `synapse.events.EventBase` and the concrete
`FrozenEvent`, `FrozenEventV2`, `FrozenEventV3`, `FrozenEventV4`, and
`FrozenEventVMSC4242` Python classes with a single Rust-backed
`Event`, exposed via `synapse.synapse_rust.events.Event`. `EventBase`
becomes a `TypeAlias` for `Event` so that the existing type annotations
across the codebase keep working.
Notable behavioural notes:
- `make_event_from_dict()` now constructs the Rust class. Event IDs for
v3+ formats are computed in the constructor (instead of lazily on
first access).
- `clone_event()` is now a single `event.deep_copy()` call. The old
shallow copy of `unsigned` was effectively a deep copy in practice;
`deep_copy()` matches that.
- The third-party event-rules callback no longer needs to call
`event.freeze()` — Events are immutable from Python by construction.
- A small `assert_never` is added in `events_worker.py` to make the
`redact_behaviour` switch exhaustive now that the type checker can
see all branches.
All test fixtures that constructed `FrozenEventV3` etc. directly are
updated to construct `Event` instead.
Adds a single `Event` Rust pyclass that replaces the Python EventBase /
FrozenEventV{1,2,3,4,VMSC4242} hierarchy. The class is added but not yet
wired into Python — callers continue to use the existing Python classes
in this commit; the migration follows in the next commit.
The internals use an `FormattedEvent` over
`EventFormatV{1,2V3,4,VMSC4242}` structs sharing an `EventCommonFields`.
Format-specific behaviour (prev_event_ids, auth_event_ids, room_id
derivation for v12 create events, etc) is encapsulated per variant.
Event IDs are computed in the constructor for v3+ formats; v1/v2 use the
`event_id` field as-is.
Two supporting Rust modules are added at the same time:
- `events::constants` — string constants for event types, top-level
fields, and per-event-type content fields, used to keep the redaction
rules and field accessors readable.
- `events::utils` — `redact()`, `compute_event_reference_hash()`, and
`calculate_event_id()`, ported from `synapse.crypto.event_signing` /
`synapse.events.utils`.
Small prerequisites for porting the Python EventBase hierarchy to Rust:
- duration: make `from_milliseconds` const and add an `IntoPyObject` impl
for owned `SynapseDuration`, so the new Rust `Event.sticky_duration()`
can return one directly to Python.
- internal_metadata: rename `copy()` to `deep_copy()` (matching the new
naming used by the rest of the events module) and make `new()` callable
from sibling modules.
- json_object: expose `object` as a `pub` field and add a `get_field`
helper so the new Event class can read from it without going through
Python.
- signatures, unsigned: add `deep_copy()` methods so the new Event class
can implement its own deep-copy.
When we port the `Event` class to Rust, the constructor will check for
the existence of required fields. To support that, we tidy up the test
code where we construct fake events to add all the required fields.
There should be no behavioural changes.
Review commit-by-commit.
The reason for the change is to make it easier to support these checks
when porting event class to Rust.
Previously, code that needed to access `prev_state_events` had to
combine a `room_version.msc4242_state_dags` boolean check with an
`isinstance(event, FrozenEventVMSC4242)` cast (or `cast()`) for the type
checker. Introduce `supports_msc4242_state_dag()` in a new
`synapse/events/py_protocol.py` which does both in one step via
`TypeIs[MSC4242Event]`, removing the need to import the concrete
`FrozenEventVMSC4242` class at every call site.
`MSC4242Event` is an `EventBase` subclass used purely for type narrowing
— it's marked with a metaclass that rejects `isinstance()` to make
accidental runtime use loud.
No behavioural change: callers continue to gate on the same room version
flag and access the same `prev_state_events` attribute.
This is in prep for using the room versions more from Rust.
Main changes:
- Change it so each room version is defined as a delta to the last one.
This is a cosmetic change that makes it easier to ensure the room
version definitions are correct (as they're defined as deltas from
previous versions).
- Move constants to `RoomVersion` constants, like `RoomVersion::V1`, for
convenience.
- Change visibility of various attributes.
Convert `prev_state_events` to use `StrCollection` rather than requiring
it to be a mutable list. None of the usages require it to be a proper
list, and besides, events are immutable and therefore so should
`event.prev_state_events`.
Based on #19708.
This is on the path to porting the entire event class to Rust, as
`event.content` will then return the new Rust class `JsonObject`.
This PR adds a pure Rust `JsonObject` class that is a `Mapping`
representing a json-style object. It uses `serde_json::Value` as its
in-memory representation and `pythonize` for conversion when a field is
looked up on the object.
I'm not thrilled with the name, but couldn't think of a better one.
This also adds `JsonObject` handling to the JSON serialisation functions
we use, as well as to the `freeze(..)` function.
Reviewable commit-by-commit.
Better to retry more quickly than have workers wait around. 5 seconds is
still a reasonable gap in time to not overwhelm anything.
This matters most in cross-worker scenarios. When locks are on the same
worker, when the lock holder releases, we signal to other locks (with
the same name/key) that they should try reacquiring the lock
immediately. But locks on other workers only re-check based on their
retry `_timeout_interval`.
Updating to 5 seconds to match the previous intentions based on the
[flawed
code](https://github.com/element-hq/synapse/blob/6100f6e4f7fb0c72f1ae2802683ebc811c0e3a77/synapse/handlers/worker_lock.py#L278).
We can assume they were trying to have 5 seconds as the max value to
retry.
Spawning from
https://github.com/element-hq/synapse/pull/19394#discussion_r3168458070
Fixes the symptoms of https://github.com/element-hq/synapse/issues/19315
/ https://github.com/element-hq/synapse/issues/19588 but not the
underlying reason causing the number to grow so large in the first
place.
```
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
```
Copied from the original pull request on [Famedly's Synapse
repo](https://github.com/famedly/synapse/pull/221) (with some edits):
Basing the time interval around a 5 seconds leaves a big window of
waiting especially as this window is doubled each retry, when another
worker could be making progress but can not.
Right now, the retry interval in seconds looks like `[0.2, 5, 10, 20,
40, 80, 160, 320, (continues to double)]` after which logging should
start about excessive times and (relatively quickly) end up with an
extremely large retry interval with an unrealistic expectation past the
heat death of the universe. 1 year in seconds = 31,536,000.
With this change, retry intervals in seconds should look more like:
```
[
0.2,
0.4,
0.8,
1.6,
3.2,
6.4,
12.8,
25.6,
51.2,
60, < never goes higher than this
]
```
Logging about excessive wait times will start at 10 minutes.
<details>
<summary>Previous breakdown when we were using 15 minutes</summary>
```
[
0.2,
0.4,
0.8,
1.6,
3.2,
6.4,
12.8,
25.6,
51.2,
102.4, # 1.7 minutes
204.8, # 3.41 minutes
409.6, # 6.83 minutes
819.2, # 13.65 minutes < logging about excessive times will start here, 13th iteration
900, # 15 minutes < never goes higher than this
]
```
</details>
Further suggested work in this area could be to define the cap, the
retry interval starting point and the multiplier depending on how
frequently this lock should be checked. See data below for reasons why.
Increasing the jitter range may also be a good idea
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
(cherry picked from commit 3f58bc50df)
Similar to #19706, let's port the `unsigned` field into a Rust class.
This does change things a bit in that we now define exactly what
unsigned fields that are allowed to be added to an event, and what
actually gets persisted. This should be a noop though, as we carefully
filter out what unsigned fields we allow in from federation, for example
As a side effect of this cleanup, I think this fixes handling
`unsigned.age` on events received over federation.
This is another stepping stone in porting the event class fully to Rust.
The new `Signatures` class is relatively simple, as we actually don't
interact with it that much in the code. It does *not* implement
`Mapping` or `MutableMapping` as that takes quite a lot of effort that
we don't need, even though it would be more ergonomic.
Fixes the symptoms of https://github.com/element-hq/synapse/issues/19315
/ https://github.com/element-hq/synapse/issues/19588 but not the
underlying reason causing the number to grow so large in the first
place.
```
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
```
Copied from the original pull request on [Famedly's Synapse
repo](https://github.com/famedly/synapse/pull/221) (with some edits):
Basing the time interval around a 5 seconds leaves a big window of
waiting especially as this window is doubled each retry, when another
worker could be making progress but can not.
Right now, the retry interval in seconds looks like `[0.2, 5, 10, 20,
40, 80, 160, 320, (continues to double)]` after which logging should
start about excessive times and (relatively quickly) end up with an
extremely large retry interval with an unrealistic expectation past the
heat death of the universe. 1 year in seconds = 31,536,000.
With this change, retry intervals in seconds should look more like:
```
[
0.2,
0.4,
0.8,
1.6,
3.2,
6.4,
12.8,
25.6,
51.2,
60, < never goes higher than this
]
```
Logging about excessive wait times will start at 10 minutes.
<details>
<summary>Previous breakdown when we were using 15 minutes</summary>
```
[
0.2,
0.4,
0.8,
1.6,
3.2,
6.4,
12.8,
25.6,
51.2,
102.4, # 1.7 minutes
204.8, # 3.41 minutes
409.6, # 6.83 minutes
819.2, # 13.65 minutes < logging about excessive times will start here, 13th iteration
900, # 15 minutes < never goes higher than this
]
```
</details>
Further suggested work in this area could be to define the cap, the
retry interval starting point and the multiplier depending on how
frequently this lock should be checked. See data below for reasons why.
Increasing the jitter range may also be a good idea
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
This comes from
https://github.com/erikjohnston/rust-signed-json/blob/main/src/json.rs.
We need to be able to serialise canonical JSON in Rust to be able to
calculate event IDs once we port the event class to Rust.
We could instead make the above a properly published crate, but feels
easier to pull it into Synapse utils.
Erik Johnston