6a63f0dcd7
Migrate dev dependencies to PEP 735 dependency groups ( #19490 )
...
This moves the dev dependencies to PEP 735 dependency groups, to help us
move to standard project metadata, which will help us moving to `uv`
(#19566 )
This requires poetry 2.2.0
2026-03-17 14:45:28 +00:00
e30001883c
Add in-repo Complement test to sanity check Synapse version matches git checkout ( #19476 )
...
This way we actually detect problems like
https://github.com/element-hq/synapse/pull/19475 as they happen instead
of being invisible until something breaks.
Sanity check that Complement is testing against your code changes
(whether it be local or from the PR in CI).
```
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo -run TestSynapseVersion
```
2026-03-11 15:30:32 -05:00
a1e9abc7df
Add Prometheus HTTP service discovery endpoint for easy discovery of all workers in Docker image ( #19336 )
...
Add Prometheus [HTTP service discovery](https://prometheus.io/docs/prometheus/latest/http_sd/ )
endpoint for easy discovery of all workers in Docker image.
Follow-up to https://github.com/element-hq/synapse/pull/19324
Spawning from wanting to [run a load
test](https://github.com/element-hq/synapse-rust-apps/pull/397 ) against
the Complement Docker image of Synapse and see metrics from the
homeserver.
`GET http://<synapse_container>:9469/metrics/service_discovery`
```json5
[
{
"targets": [ "<host>", ... ],
"labels": {
"<labelname>": "<labelvalue>", ...
}
},
...
]
```
The metrics from each worker can also be accessed via
`http://<synapse_container>:9469/metrics/worker/<worker_name>` which is
what the service discovery response points to behind the scenes. This
way, you only need to expose a single port (9469) to access all metrics.
<details>
<summary>Real HTTP service discovery response</summary>
```json5
[
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "event_persister",
"index": "1",
"__metrics_path__": "/metrics/worker/event_persister1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "event_persister",
"index": "2",
"__metrics_path__": "/metrics/worker/event_persister2"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "background_worker",
"index": "1",
"__metrics_path__": "/metrics/worker/background_worker1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "event_creator",
"index": "1",
"__metrics_path__": "/metrics/worker/event_creator1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "user_dir",
"index": "1",
"__metrics_path__": "/metrics/worker/user_dir1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "media_repository",
"index": "1",
"__metrics_path__": "/metrics/worker/media_repository1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "federation_inbound",
"index": "1",
"__metrics_path__": "/metrics/worker/federation_inbound1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "federation_reader",
"index": "1",
"__metrics_path__": "/metrics/worker/federation_reader1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "federation_sender",
"index": "1",
"__metrics_path__": "/metrics/worker/federation_sender1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "synchrotron",
"index": "1",
"__metrics_path__": "/metrics/worker/synchrotron1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "client_reader",
"index": "1",
"__metrics_path__": "/metrics/worker/client_reader1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "appservice",
"index": "1",
"__metrics_path__": "/metrics/worker/appservice1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "pusher",
"index": "1",
"__metrics_path__": "/metrics/worker/pusher1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "device_lists",
"index": "1",
"__metrics_path__": "/metrics/worker/device_lists1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "device_lists",
"index": "2",
"__metrics_path__": "/metrics/worker/device_lists2"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "stream_writers",
"index": "1",
"__metrics_path__": "/metrics/worker/stream_writers1"
}
},
{
"targets": [
"localhost:9469"
],
"labels": {
"job": "main",
"index": "1",
"__metrics_path__": "/metrics/worker/main"
}
}
]
```
</details>
And how it ends up as targets in Prometheus
(http://localhost:9090/targets ):
(image)
### Testing strategy
1. Make sure your firewall allows the Docker containers to communicate
to the host (`host.docker.internal`) so they can access exposed ports of
other Docker containers. We want to allow Synapse to access the
Prometheus container and Grafana to access to the Prometheus container.
- `sudo ufw allow in on docker0 comment "Allow traffic from the default
Docker network to the host machine (host.docker.internal)"`
- `sudo ufw allow in on br-+ comment "(from Matrix Complement testing)
Allow traffic from custom Docker networks to the host machine
(host.docker.internal)"`
- [Complement firewall
docs](https://github.com/matrix-org/complement/blob/ee6acd9154bbae2d0071a9cb39593c0a5e37268b/README.md#potential-conflict-with-firewall-software )
1. Build the Docker image for Synapse: `docker build -t
matrixdotorg/synapse -f docker/Dockerfile . && docker build -t
matrixdotorg/synapse-workers -f docker/Dockerfile-workers .`
([docs](https://github.com/element-hq/synapse/blob/7a24fafbc376b9bffeb3277b1ad4aa950720c96c/docker/README-testing.md#building-and-running-the-images-manually ))
1. Start Synapse:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
-e SYNAPSE_REPORT_STATS=no \
-e SYNAPSE_ENABLE_METRICS=1 \
-p 8008:8008 \
-p 9469:9469 \
matrixdotorg/synapse-workers:latest
```
- Also try with workers:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
-e SYNAPSE_REPORT_STATS=no \
-e SYNAPSE_ENABLE_METRICS=1 \
-e SYNAPSE_WORKER_TYPES="\
event_persister:2, \
background_worker, \
event_creator, \
user_dir, \
media_repository, \
federation_inbound, \
federation_reader, \
federation_sender, \
synchrotron, \
client_reader, \
appservice, \
pusher, \
device_lists:2, \
stream_writers=account_data+presence+receipts+to_device+typing" \
-p 8008:8008 \
-p 9469:9469 \
matrixdotorg/synapse-workers:latest
```
1. You should be able to see Prometheus service discovery endpoint at
http://localhost:9469/metrics/service_discovery
1. Create a Prometheus config (`prometheus.yml`)
```yaml
global:
scrape_interval: 15s
scrape_timeout: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: synapse
scrape_interval: 15s
metrics_path: /_synapse/metrics
scheme: http
# We set `honor_labels` so that each service can set their own `job`
label
#
# > honor_labels controls how Prometheus handles conflicts between
labels that are
# > already present in scraped data and labels that Prometheus would
attach
# > server-side ("job" and "instance" labels, manually configured target
# > labels, and labels generated by service discovery implementations).
# >
# > *--
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config *
honor_labels: true
# Use HTTP service discovery
#
# Reference:
# - https://prometheus.io/docs/prometheus/latest/http_sd/
# -
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config
http_sd_configs:
- url: 'http://localhost:9469/metrics/service_discovery '
```
1. Start Prometheus (update the volume bind mount to the config you just
saved somewhere):
```
docker run \
--detach \
--name=prometheus \
--add-host host.docker.internal:host-gateway \
-p 9090:9090 \
-v
~/Documents/code/random/prometheus-config/prometheus.yml:/etc/prometheus/prometheus.yml
\
prom/prometheus
```
1. Make sure you're seeing some data in Prometheus. On
http://localhost:9090/query , search for `synapse_build_info`
1. Start [Grafana](https://hub.docker.com/r/grafana/grafana )
```
docker run -d --name=grafana --add-host
host.docker.internal:host-gateway -p 3000:3000 grafana/grafana
```
1. Visit the Grafana dashboard, http://localhost:3000/ (Credentials:
`admin`/`admin`)
1. **Connections** -> **Data Sources** -> **Add data source** ->
**Prometheus**
- Prometheus server URL: `http://host.docker.internal:9090 `
1. Import the Synapse dashboard:
https://github.com/element-hq/synapse/blob/develop/contrib/grafana/synapse.json
2026-01-14 18:02:55 -06:00
9dae6cc595
Add a way to expose metrics from the Docker image (SYNAPSE_ENABLE_METRICS) ( #19324 )
...
Spawning from wanting to [run a load
test](https://github.com/element-hq/synapse-rust-apps/pull/397 ) against
the Complement Docker image of Synapse and see metrics from the
homeserver.
### Why not just provide your own homeserver config?
Probably possible but it gets tricky when you try to use the workers
variant of the Docker image (`docker/Dockerfile-workers`). The way to
workaround it would probably be to `yq` edit everything in a script and
change `/data/homeserver.yaml` and `/conf/workers/*.yaml` to add the
`metrics` listener. And then modify `/conf/workers/shared.yaml` to add
`enable_metrics: true`. Doesn't spark much joy.
2026-01-01 14:00:00 -06:00
728512918e
Exclude .lock file from /usr/local when building docker images ( #19107 )
2025-10-30 10:17:35 +00:00
67f22a200d
Update Docker images to use Debian trixie (13) and thus Python 3.13 ( #19064 )
2025-10-20 16:49:17 +01:00
8f61bdb470
Note optional Element Commecial License in SPDX specifiers ( #18973 )
2025-09-26 12:43:07 +01:00
7c32988f6b
Update URLs in dockerfile metadata ( #18971 )
2025-09-26 12:40:50 +01:00
f638a76ba4
Avoid relying on rsync during Docker build ( #18287 )
...
Use targeted COPY commands instead of rsync to avoid having a symlinked
/lib as the destination of a COPY (which buildkit does not support).
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
2025-04-01 12:32:34 +00:00
bd08a01fc8
Dockerfile: set package arch via APT config option ( #18271 )
2025-03-25 13:58:40 +00:00
33bcef9dc7
Update Poetry to 2.1.1, including updating the lock file version. ( #18251 )
2025-03-21 15:32:52 +00:00
1525a3b4d4
Speedup the building of Docker images ( #18038 )
...
This is a split off #18033
This uses a few tricks to speed up the building of docker images:
- This switches to use `uv pip install` instead of `pip install`. This
saves a bunch of time, especially when cross-compiling
- I then looked at what packages were not using binary wheels: I
upgraded MarkupSafe to have binaries for py3.12, and got back to Python
3.12 because hiredis didn't have builds for py3.13 with the version we
were using
- The generation of the requirements.txt is arch-agnostic, so I've
switched this one to run on the build architecture, so that both arch
can share it
- The download of runtime depdendencies can be done on the build
architecture through manual `apt-get download` plus `dpkg --extract`
- We were using -slim images, but still installed a bunch of -dev
dependencies. Turns out, all the dev dependencies were already installed
in the non-slim image, which saves a bunch of time as well
2025-02-19 10:55:15 +00:00
1c2b18a704
Bump Synapse Dockerfile default to Python 3.12 ( #17887 )
2024-11-05 13:15:10 +00:00
5871daf877
Use consistent casing between FROM and AS ( #17431 )
...
Signed-off-by: Joe Groocock <me@frebib.net >
2024-07-11 13:56:25 +01:00
7254015665
Correct licensing metadata on the Docker image ( #17141 )
...
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
2024-05-01 16:23:42 +01:00
8613f7693e
More renaming
2023-12-13 15:41:11 +00:00
3cf1a3aa17
Use bookwork as docker base image ( #16324 )
2023-09-15 13:14:10 +01:00
874378c052
Docker fully qualified image names ( #15689 )
...
* Fully qualified docker image names for the main Dockerfile and Complement related.
* Fully qualified docker image names for Dockerfiles associated with building Debian release artifacts.
This one is harder and is separate from the other commit in case it wasn't correct or was unwanted. I decided to
do the expansion on the docker images in the Dockerfile itself, instead of the various source places that build
which distribution that is selected, as it would have been more invasive with the scripts breaking up the string
for tagging and such. This one is untested.
* Changelog
* Update docker/Dockerfile-workers
* Update docker/complement/Dockerfile
---------
Co-authored-by: reivilibre <olivier@librepush.net >
2023-05-31 15:13:31 +00:00
08297f2f18
Add pkg-config package to Stage 0 ( #15567 )
2023-05-12 11:32:09 +02:00
d4eba4409f
Install rust during Stage 0 of docker build ( #15239 )
...
* Install rust during Stage 0 of docker build
Thanks to @atomdmac for spotting the fix.
Fixes #15179 .
* Changelog
2023-03-13 12:12:02 +00:00
cf18fea9e1
Dockerfile: Bump Python version from 3.9 to 3.11 ( #14875 )
...
Closes https://github.com/matrix-org/synapse/issues/13234
Signed-off-by: Katia Esposito <1695469+katlol@users.noreply.github.com >
Signed-off-by: Katia Esposito <1695469+katlol@users.noreply.github.com >
2023-01-20 12:07:13 +00:00
b88cfe6d41
Require poetry>=1.3.2 ( #14860 )
...
* Upgrade to new lockfile format
Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write.
Cheat sheet:
```
poetry --version
poetry show > scratch/before
pipx upgrade poetry
poetry --version
poetry show > scratch/after
diff scratch{before,after} && echo "no change!"
```
* Use Poetry 1.3.2 when reading or writing lockfile
* Remove unneeded(?) poetry dep for cibuildwheel
* Update docs
* Remove redundant call to setup-python
* Remove outdated comments related to Poetry 1.x
* Remove outdated docs line
was fixed in #13082
* Minor improvements to poetry cheat sheet
* Invoke setup-python-poetry with explicit version
Not sure about this. It's hardcoding versions everywhere.
* Changelog
* Check the lockfile is version 2.0
Might one day incorporate other checks like #14742
* Typo fixes, thanks Sean
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com >
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com >
2023-01-17 18:04:44 +00:00
eb2defc2f7
Add release note and update doc regarding ICU ( #14712 )
...
Fixes #14704 .
Signed-off-by: Mathieu Velten <mathieuv@matrix.org >
2022-12-20 16:06:26 +00:00
2a3cd59dd0
Add optional ICU support for user search ( #14464 )
...
Fixes #13655
This change uses ICU (International Components for Unicode) to improve boundary detection in user search.
This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
2022-12-12 13:21:17 +01:00
22e91b8019
docker: remove useless cargo install with apt ( #14636 )
...
Signed-off-by: Mathieu Velten <mathieuv@matrix.org >
2022-12-07 15:29:32 +01:00
6fee2f49f3
Cache Rust build cache when building docker images ( #14130 )
2022-10-17 17:21:14 +00:00
bc2bd92b93
Merge remote-tracking branch 'origin/release-v1.69' into develop
2022-10-14 14:11:27 -04:00
29ee4b6698
Fix docker build OOMing in CI for arm64 builds ( #14173 )
...
Co-authored-by: David Robertson <davidr@element.io >
2022-10-13 18:16:21 +00:00
3f057e4c54
Use minimal Rust installation in docker images and CI ( #14141 )
2022-10-12 10:47:02 +01:00
d94bcbced3
Fix pinning Rust deps in docker images ( #14129 )
2022-10-11 10:53:34 +00:00
d5292b8017
Fix Docker build when Rust .so has been build locally first ( #13811 )
...
Signed-off-by: Mathieu Velten <mathieuv@matrix.org >
2022-09-16 15:38:54 +00:00
c9b7e97355
Add a stub Rust crate ( #12595 )
2022-09-06 19:01:37 +01:00
18e4092801
Bump docker image to use stable poetry version ( #13688 )
2022-09-01 13:46:47 +01:00
503a95804e
Install cryptography build dependencies in requirements image. ( #13372 )
2022-08-03 11:16:32 +01:00
05aeeb3a80
Enable Complement CI tests in the 'latest deps' test run. ( #13213 )
...
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com >
2022-08-01 10:55:31 +00:00
e9ce4d089b
Use and recommend poetry 1.1.14, up from 1.1.12 ( #13285 )
2022-07-15 16:18:47 +01:00
c04e25789e
Enable Complement testing in the 'Twisted Trunk' CI runs. ( #13079 )
2022-07-01 15:42:49 +00:00
538044ac01
Collapse Docker build commands in Complement CI runs to make the logs easier to read. ( #13058 )
2022-06-15 14:42:27 +00:00
f5b1c09909
Pin poetry.core in Docker images ( #12853 )
2022-05-24 11:35:08 +01:00
706456de1f
Mark Dockerfile as requiring BuildKit ( #12541 )
...
Co-authored-by: David Robertson <davidr@element.io >
2022-04-26 15:31:52 +01:00
e5a76ec00b
Dump setuptools; correct pyproject version number ( #12478 )
2022-04-20 17:33:20 +01:00
5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes ( #12439 )
...
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.
When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true
> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205 Twisted[tls]>=18.7.0 from https://files.pythonhosted.org/packages/db/99/38622ff95bb740bcc991f548eb46295bba62fcb6e907db1987c4d92edd09/Twisted-22.4.0-py3-none-any.whl#sha256=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```
The underlying pip issue is https://github.com/pypa/pip/issues/9644 . A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.
(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311 , but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
3a7e97c7ad
Poetry: use locked environment in Docker images ( #12385 )
2022-04-07 11:43:31 +00:00
4aeb00ca20
Move synctl into synapse._scripts and expose as an entrypoint ( #12140 )
2022-03-04 11:58:49 +00:00
f3f0ab10fe
Move scripts directory inside synapse, exposing as setuptools entry_points ( #12118 )
...
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com >
2022-03-02 13:00:16 +00:00
f26e390a40
Use Python 3.9 in Synapse dockerfiles by default ( #12112 )
2022-03-01 13:55:18 +00:00
5598556b77
Docker: remove VOLUME directive ( #11997 )
...
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.
I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.
That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.
So... this wasn't doing anything, except annoying Complement?
2022-02-15 13:59:15 +00:00
d41c4654db
Use buildkit's cache feature to speed up docker builds ( #11691 )
...
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
d5305000f1
Docker healthcheck timings - add startup delay and changed interval ( #9913 )
...
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
2021-05-05 17:33:04 +02:00
cb657eb2f8
Put opencontainers labels to the final image ( #9765 )
...
They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.
Signed-off-by: Johannes Wienke <languitar@semipol.de >
2021-04-08 13:49:14 +01:00
Quentin Gliech