Ok so this is pretty shit, i hate the integration test suite now.
The reason why we return the test functions with `as any` in the
hello test is because we had to remove `Record<string, any>` from
mocha's test context interface, otherwise the interface would
have been completely useless. Maybe there is a ts setting though
to not infer any from `this` at all? and just ignore those properties.
The tsconfig.json situation is a bit weird, i don't understand why
it's in this situation. However, it seems like we can try to
https://github.com/jaredpalmer/tsdx/issues/84#issuecomment-489690504
use this workaround so that ts language features work in the test
directory.
I think we should focus on doing as little effort as possible getting
these tests into working condition. If something is too complicated,
it will need removing. If we need to make additional tests,
this entire integration tests directory should be moved
to a legacy-integration directory and we can start afresh.
We should also ideally not integration tests as much as possible
and try to reuse the unit helpers from MPS.
This is even going to be critical later on.
This was introduced in https://github.com/Gnuxie/Draupnir/pull/54/ (and therefore 1.83.0). Essentially we forgot to remove the room from the protected rooms set, when the remove command was used.
Ontop of this something to note is that during testing it is clear that the protected rooms set is loaded when configuring mjolnir, not when starting it. This is problematic as it means setup code in `fixtures.ts` does not actually wipe the protected rooms set.
* Experimental Protection to propagate room level bans to policies.
- Needs an automated option
- I really want this to be enabled by default
- It needs to be easily configurable and very visible because it's a really useful feature.
- Need to check that they are not already banned on a policy list.
- Allow possibility to rely last message like a report behind spoiler text.
* Use MatrixDataManager for enabled protections.
This will allow us to create "enabled by default" protections
via a schema migration.
* Enable BanPropagationProtection by default
* BanPropagation: only prompt when user is not already banned.
* Test for BanPropagationProtection.
* clearTimeout for prompt reactions if we got a reaction.
* Allow renderMatrixAndSend to not need a reply.
* document getFirstEventMatching
* basic ban conversion, but i have better ideas
* Still very WIP on CLIM prompt-for-accept semantics.
* Introduce promotable streams.
This allows parameters to specify details to prompt for missing
arguments
and allow for interactive commands.
* Changes that were made before PolicyListManager that no longer make sense
We don't want the default list anymore since we're just going to prompt
with the lists that they can choose from.
* Fix semantics of TagDynamicEnvironment.
Bind and write were wrong and bind was binding to the node name
instead of the variable name.
* The JSX factory can render presentation types to DocumentNodes, unsure if this is the right
move yet but it works
* Attributes for anchor nodes now render properly
* Ban command prompts are working!!!!
* Stub AppserviceBotEmitter.
There isn't much we can do right now until there is time to work on
https://github.com/Gnuxie/Draupnir/issues/13.
* Combine ban/unban syntax.
* Remove old UnbanBanCommands.
WARNING: There is a major difference in that the ban command no longer supports
globs, I don't think?
* Activate new unban command.
* The presentation type boolean will have to be just a string for now.
I don't think it makes sense to read them into actual booleans.
* configurable defaults for ban reason.
This started out as just a way to find out why mjolnir was syncing with lists several times for each update to a policy list.
The main changes are
- Verbosity was irrelevant to the sync command but for some reason was an option.
Unfortunately all this did was suppress whether to tell you when it had finished, meaning it wouldn't
when verbose logging was disabled. Historically this was probably a parameter that got passed through
to applyServerAcl/applyUserBans, which can be horribly verbose, but they access the config directly.
- Stop emitting `'PolicyList.update'` when there are no changes.
- Include a revision ID for the `'PolicyList.update'`method and event.
- Use the revision ID in the `ProtectedRoomsSet` so that we don't unnecessarily resynchronize all rooms when the `'PolicyList.update'` event is received. Though not when the `sync` command is used. Since this is supposed to `sync` in the case when there is a state reset or otherwise or the user has changed some room settings.
- insert an await lock around the `PolicyList.update` method to avoid a race condition where a call can be started and finished within the extent of an existing call (via another task, this can happen if the server is slow with handling one request). `PolicyList.udpate` now has a helper that is synchronous to be called directly after requesting the room state. The reason for this is to enforce that no one `await`s while updating the policy list's cache of rules. Which is important because it is one of the biggest methods that I tolerate and visually checking for `await` is impossible.
- The revision ID uses a ULID, but this is unnecessary and could have just been a "dumb counter".
closes https://github.com/matrix-org/mjolnir/issues/447
* Refactor Matrix event listener in Mjolnir and ManagedMjolnir.
closes https://github.com/matrix-org/mjolnir/issues/411.
Issue #411 says that we have to be careful about room.join,
but this was before we figured how to make matrix-appservice-bridge
echo events sent by its own intents.
* Remove MatrixClientListener since it isn't actually needed.
* Protect which config values can be used for ManagedMjolnirs.
* Introduce MatrixSendClient
so listeners aren't accidentally added to a MatrixClient instead
of MatrixEmitter.
* doc
* Move provisioned mjolnir config to src/config.
This just aids maintance so whenever someone goes to change the config
of the bot they will see this and update it.
* doc for matrix intent listener.
The Sentry package is very useful for monitoring runtime errors. With this PR,
we simply add the necessary mechanism to:
- log to sentry any uncaught error that reaches the toplevel, including startup errors.
The ACL unit allows you to combine an policy lists and conveniently test users and servers against them.
The main motivation for this work is provide access control on who can provision and continue to use mjolnir instances in the appservice component.
We include a new recommendation type org.matrix.mjolnir.allow which can be used with user and server entity types to create allow lists.
We have also replaced the destructing of policy lists in applyServerACL and applyMemberBans (in ProtectedRooms.ts) with calls to the AccessControlUnit.
Adding commands to add/remove allowed entities is not something i want to do at the moment.
In both our instructions and our tests, we use the r0 endpoint to intercept abuse reports. This endpoint is deprecated and not implemented by all clients. This PR updates the instructions and tests to the new endpoint.
* Attempt to factor out protected rooms from Mjolnir.
This is useful to the appservice because it means we don't
have to wrap a Mjolnir that is designed to sync.
It's also useful if we later on want to have specific
settings per space.
It's also just a nice seperation between Mjolnir's needs while
syncing via client-server and the behaviour of syncing policy rooms.
### Things that have changed
- `ErrorCache` no longer a static class (phew), gets used by `ProtectedRooms`.
- `ManagementRoomOutput` class gets created to handle logging back to the management room.
- Responsibilities for syncing member bans and server ACL are handled by `ProtectedRooms`.
- Responsibilities for watched lists should be moved to `ProtectedRooms` if they haven't been.
- `EventRedactionQueue` is moved to `ProtectedRooms` since this needs to happen after
member bans.
- ApplyServerAcls moved to `ProtectedRooms`
- ApplyMemberBans move to `ProtectedRooms`
- `logMessage` and `replaceRoomIdsWithPills` moved to `ManagementRoomOutput`.
- `resyncJoinedRooms` has been made a little more clear, though I am concerned about how often it does run because it does seem expensive.
* ProtectedRooms is not supposed to track joined rooms.
The reason is because it is supposed to represent a specific
set of rooms to protect, not do horrible logic
for working out what rooms mjolnir is supposed to protect.
* Rework the banning and unbanning of entities in PolicyLists.
1. We keep track of the event that created a list rule so that we
can remove the rule by having a way to determine the original state key for the rule.
This is because the state key of rules can be anything and should not be
relied on by Mjolnir to unban things (which it was doing).
2. The old scheme for producing a state key was causing for some entities to escape bans
https://github.com/matrix-org/mjolnir/issues/322.
We could have used a hash or something similar, but we know that
the reason for the `rule:${entity}` scheme existed was for ease of debugging
and finding rules in devtools. So instead we have followed a scheme simalar to
bridges where the first character of an mxid is replaced with an underscore.
Everything else just gets put into the state key. Since domains can't have '@'
and room ids, aliases can't either.
3. We have stopped the need for Mjolnir to wait for the next response from sync after banning,
unbanning an entity so that we can apply ACL's sooner.
* Use PolicyList's `banEntity` method to create imported rules.
If you add `"no-floating-promises": true` it's very easy
to find where this is done accidentally.
Not sure we can keep that on all the time yet though..
Replace acceptInvitesFromGroup with acceptInvitesFromSpace.
https://github.com/matrix-org/mjolnir/issues/125https://github.com/matrix-org/mjolnir/issues/99
acceptInvitesFromGroup was implemented with an experimental api
that was a precursor to spaces which was refereed to
as either communities or groups.
Support for communities/groups ended in Synapse 1.61.0
https://github.com/matrix-org/synapse/releases/tag/v1.61.0.
To test we just edit the config dynamically which changes how the join room listener functions
though idk, shouldn't we have just made a new mjolnir instance
for this test, or changed the config before the test started somehow?
Co-authored-by: jesopo <github@lolnerd.net>
The implementation is rubbish, as it doesn't avoid the exponential backoff
Remove default rate limit testing.
It doesn't work. No there really isn't more to say about it
you're welcome to dispute it if you're going to do the work investigating. I'm not.
We used to have a test here that tested whether Mjolnir was going to carry out a redact order the default limits in a reasonable time scale.
Now I think that's never going to happen without writing a new algorithm for respecting rate limiting.
Which is not something there is time for.
https://github.com/matrix-org/synapse/pull/13018
Synapse rate limits were broken and very permitting so that's why the current hack worked so well.
Now it is not broken, so our rate limit handling is.
https://github.com/matrix-org/mjolnir/commit/b850e4554c6cbc9456e23ab1a92ede547d044241
Honestly I don't think we can expect anyone to be able to use Mjolnir under default rate limits.
well, it's not quite simple as broken, but it is broken. With the default level in synapse (which is what matrix.org uses) it is struggling to redact 15 messages within 5 minutes. that means 5 messages over the burst count. This is ofc ontop mjolnir sending reactions / responding to replies (which isn't much but... enough to mess with the rate limiter since ofc, Synapse tells requests to wait x amount of time before trying again, but that doesn't help for concurrent requests since ofc there's only 1 slot available at that future time. This means Synapse just wacks everything with exponentially longer shit without many (or any?) events going through
it used to be fine
because rate limiting in synapse used to be a lot more liberal because it was "broken" or something, that's not me saying it's broken that's just what synapse devs say which is probably true.
if all requests went into a queue then yeah you could eliminate one problem
but that's a lot of work and i don't think we should be doing it
cos no one uses mjolnir like this anyways
* Stop the config being global (in almost all contexts).
* make sure unit test has a config
* Make failing word list more visible
* Only use Healthz from index.ts
Not really sure how useful it is anyways?
* banListTest would applyACL before rules appeared in `/state`.
Mjolnir will call applyServerAcls several times while a policy list is being updated, sometimes concurrently. This means a request to set a server ACL in a room which has old data can finish after a more recent recent request with the correct ACL. This means that the old ACL gets applied to the rooms (for a moment).
This is a follow up from https://github.com/matrix-org/mjolnir/pull/314/commits/551065815e65e6117d7e8105afd6f5da9b87c8af
* Only allow one invocation of applyServerAcls at a time as to not conflict with each other by using a promise chain.
We don't use the throttle queue because we don't want to be blocked by other background tasks.
We don't make another throttle queue because we don't want throttling and we don't want to delay the ACL application, which can happen even with throttle time of 0.
Towards opinions in PolicyLists.
This changeset is part of an ongoing effort to implement "opinions"
within policy lists, as per MSC3847.
For the time being:
- we rename BanList into PolicyList;
- we cleanup a little dead code;
- we replace a few `string`s with `enum`;
- `ListRule` becomes an abstract class with two concrete subclasses `ListRuleBan` and `ListRuleOpinion`.
gnuxie