fe742653ff
Add an API to list changes to quarantine state of media ( #19558 )
...
Fixes https://github.com/element-hq/synapse/issues/19352
(See issue for history of this feature and previous PRs)
> First, a [naive
implementation](https://github.com/element-hq/synapse/pull/19268 ) of the
endpoint was introduced, but it quickly ran into [performance issues on
query](https://github.com/element-hq/synapse/pull/19312 ) and [long
startup times](https://github.com/element-hq/synapse/issues/19349 ),
leading to its
[removal](https://github.com/element-hq/synapse/pull/19351 ). It also
didn't actually work, and would fail to expose media when it was
"unquarantined", so a [partial
fix](https://github.com/element-hq/synapse/pull/19308 ) was attempted,
where the suggested direction is to use a
[stream](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html#cheatsheet-for-creating-a-new-stream )
instead of a timestamp column.
This PR re-introduces the API building on the previous feedback:
* Adds a stream which tracks when media becomes (un)quarantined.
* Runs a background update to capture already-quarantined media.
* Adds a new admin API to return rows from the stream table.
We track both quarantine and unquarantine actions in the stream to allow
downstream consumers to process the records appropriately. Namely, to
allow our Synapse exchange in HMA to remove hashes for unquarantined
media (use case further explained in the
[issue](https://github.com/element-hq/synapse/issues/19352 )).
**Note**: This knowingly does not capture all cases of media being
quarantined. Other call sites are lower priority for T&S, and can be
addressed in a future PR. ~~An issue will be created after this PR is
merged to track those sites.~~
https://github.com/element-hq/synapse/issues/19672
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
---------
Co-authored-by: turt2live <1190097+turt2live@users.noreply.github.com >
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com >
Co-authored-by: Eric Eastwood <erice@element.io >
2026-04-09 11:41:31 -06:00
987b61a92b
Revert "Add an Admin API endpoint for listing quarantined media ( #19268 )" ( #19351 )
...
Fixes #19349
This reverts commit 3f636386a6https://github.com/element-hq/synapse/pull/19268 ) as the DB migration
was taking too long and blocking media access while it happened.
See https://github.com/element-hq/synapse/issues/19349 for further
information.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
---------
Co-authored-by: Travis Ralston <travpc@gmail.com >
2026-01-06 21:37:23 +00:00
3f636386a6
Add an Admin API endpoint for listing quarantined media ( #19268 )
...
Co-authored-by: turt2live <1190097+turt2live@users.noreply.github.com >
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com >
2025-12-12 13:30:21 +00:00
fc244bb592
Use type hinting generics in standard collections ( #19046 )
...
aka PEP 585, added in Python 3.9
- https://peps.python.org/pep-0585/
- https://docs.astral.sh/ruff/rules/non-pep585-annotation/
2025-10-22 16:48:19 -05:00
35c9cbb09d
Add an Admin API to query a piece of local or cached remote media by ID ( #18911 )
2025-09-23 16:25:56 -05:00
5a9ca1e3d9
Introduce Clock.call_when_running(...) to include logcontext by default ( #18944 )
...
Introduce `Clock.call_when_running(...)` to wrap startup code in a
logcontext, ensuring we can identify which server generated the logs.
Background:
> Ideally, nothing from the Synapse homeserver would be logged against the `sentinel`
> logcontext as we want to know which server the logs came from. In practice, this is not
> always the case yet especially outside of request handling.
>
> Global things outside of Synapse (e.g. Twisted reactor code) should run in the
> `sentinel` logcontext. It's only when it calls into application code that a logcontext
> gets activated. This means the reactor should be started in the `sentinel` logcontext,
> and any time an awaitable yields control back to the reactor, it should reset the
> logcontext to be the `sentinel` logcontext. This is important to avoid leaking the
> current logcontext to the reactor (which would then get picked up and associated with
> the next thing the reactor does).
>
> *-- `docs/log_contexts.md`
Also adds a lint to prefer `Clock.call_when_running(...)` over
`reactor.callWhenRunning(...)`
Part of https://github.com/element-hq/synapse/issues/18905
2025-09-22 10:27:59 -05:00
a31d53b28f
Use twisted.internet.testing module in tests instead of deprecated twisted.test.proto_helpers. ( #18728 )
...
Follows: #18727
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org >
2025-07-30 12:32:10 +01:00
d17295e5c3
Store hashes of media files, and allow quarantining by hash. ( #18277 )
...
This PR makes a few radical changes to media. This now stores the SHA256
hash of each file stored in the database (excluding thumbnails, more on
that later). If a set of media is quarantined, any additional uploads of
the same file contents or any other files with the same hash will be
quarantined at the same time.
Currently this does NOT:
- De-duplicate media, although a future extension could be to do that.
- Run any background jobs to identify the hashes of older files. This
could also be a future extension, though the value of doing so is
limited to combat the abuse of recent media.
- Hash thumbnails. It's assumed that thumbnails are parented to some
form of media, so you'd likely be wanting to quarantine the media and
the thumbnail at the same time.
2025-03-27 17:26:34 +00:00
d0a474d312
Enable authenticated media by default ( #17889 )
...
Co-authored-by: Olivier 'reivilibre <oliverw@matrix.org >
2024-11-20 14:48:22 +00:00
f0d6f14047
Parse Integer negative value validation ( #16920 )
2024-04-16 19:12:36 +00:00
23740eaa3d
Correctly mention previous copyright ( #16820 )
...
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
8e1e62c9e0
Update license headers
2023-11-21 15:29:58 -05:00
ff716b483b
Return attrs for more media repo APIs. ( #16611 )
2023-11-09 11:00:30 -05:00
26b960b08b
Register media servlets via regex. ( #16419 )
...
This converts the media servlet URLs in the same way as
(most) of the rest of Synapse. This will give more flexibility
in the versions each endpoint exists under.
2023-10-06 07:22:55 -04:00
4fc8875876
Refactor media modules. ( #15146 )
...
* Removes the `v1` directory from `test.rest.media.v1`.
* Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`.
* Flatten the `v1` directory from `synapse.rest.media`, but leave compatiblity
with 3rd party media repositories and spam checkers.
2023-02-27 08:26:05 -05:00
9bb2eac719
Bump black from 22.12.0 to 23.1.0 ( #15103 )
2023-02-22 15:29:09 -05:00
d22c1c862c
Respond correctly to unknown methods on known endpoints ( #14605 )
...
Respond with a 405 error if a request is received on a known endpoint,
but to an unknown method, per MSC3743.
2023-02-09 13:04:24 -05:00
2281427175
Use literals in place of HTTPStatus constants in tests ( #13488 )
...
* Use literals in place of `HTTPStatus` constants in tests
* newsfile
* code style
* code style
2022-08-10 11:01:12 -07:00
1595052b26
Use literals in place of HTTPStatus constants in tests ( #13479 )
...
Replace
- `HTTPStatus.NOT_FOUND`
- `HTTPStatus.FORBIDDEN`
- `HTTPStatus.UNAUTHORIZED`
- `HTTPStatus.CONFLICT`
- `HTTPStatus.CREATED`
Signed-off-by: Dirk Klimpel <dirk@klimpel.org >
2022-08-09 14:56:43 +01:00
c97042f7ee
Use literals in place of HTTPStatus constants in tests ( #13469 )
2022-08-08 13:21:27 -07:00
f0b03186d9
Add type hints for tests/unittest.py. ( #12347 )
...
In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
2022-04-01 16:04:16 +00:00
e24ff8ebe3
Remove HomeServer.get_datastore() ( #12031 )
...
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
2022-02-23 11:04:02 +00:00
8428ef66c7
Add type hints to synapse/tests/rest/admin ( #11590 )
2021-12-16 14:59:56 -05:00
b3bcacf3c1
Add missing errcode to parse_string and parse_boolean ( #11542 )
2021-12-09 11:23:34 +00:00
e5f426cd54
Add type hints to synapse/tests/rest/admin ( #11501 )
2021-12-03 13:57:13 +00:00
35b1900f00
Convert status codes to HTTPStatus in tests.rest.admin ( #11455 )
2021-11-30 09:53:54 +00:00
2c61a318cc
Show error when timestamp in seconds is provided to the /purge_media_cache API ( #11101 )
2021-10-20 14:41:48 +00:00
bb7fdd821b
Use direct references for configuration variables (part 5). ( #10897 )
2021-09-24 07:25:21 -04:00
50022cff96
Add reactor to SynapseRequest and fix up types. ( #10868 )
2021-09-24 11:01:25 +01:00
bfb4b858a9
Create a constant for a small png image in tests. ( #10834 )
...
To avoid duplicating it between a few tests.
2021-09-16 12:01:14 -04:00
642a42edde
Flatten the synapse.rest.client package ( #10600 )
2021-08-17 11:57:58 +00:00
590cc4e888
Add type hints to additional servlet functions ( #10437 )
...
Improves type hints for:
* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request
And fixes any incorrect calls that resulted from unknown types.
2021-07-21 18:12:22 +00:00
0284d2a297
Add new admin APIs to remove media by media ID from quarantine. ( #10044 )
...
Related to: #6681 , #5956 , #10040
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-06-02 18:50:35 +01:00
65e6c64d83
Add an admin API for unprotecting local media from quarantine ( #10040 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-05-26 11:19:47 +01:00
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl >`
2021-04-14 15:34:27 +01:00
0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md )
- Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
42d3a28d8b
Removes unnecessary declarations in the tests for the admin API. ( #9063 )
2021-01-11 11:15:54 -05:00
394516ad1b
Remove spurious "SynapseRequest" result from `make_request"
...
This was never used, so let's get rid of it.
2020-12-15 22:35:40 +00:00
09ac0569fe
Fix broken testcase ( #8851 )
...
This test was broken by #8565 . It doesn't need to set set `self.clock`
here anyway - that is done by `setUp`.
2020-12-01 11:04:57 +00:00
acfe3b3065
Remove redundant HomeserverTestCase.render
2020-11-16 18:24:08 +00:00
129ae841e5
Make make_request actually render the request
...
remove the stubbing out of `request.process`, so that `requestReceived` also renders the request via the appropriate resource.
Replace render() with a stub for now.
2020-11-16 18:24:00 +00:00
cfd895a22e
use global make_request() directly where we have a custom Resource
...
Where we want to render a request against a specific Resource, call the global
make_request() function rather than the one in HomeserverTestCase, allowing us
to pass in an appropriate `Site`.
2020-11-15 23:09:03 +00:00
49d72dea2a
Add an admin api to delete local media. ( #8519 )
...
Related to: #6459 , #3479
Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete
a single file from server.
2020-10-26 17:02:28 +00:00
Travis Ralston