Commit Graph

1785 Commits

Author SHA1 Message Date
Evgeny Poberezkin
dd67de4d71 Merge branch 'master' into short-links 2025-03-31 16:54:07 +01:00
Evgeny
285fd93c32 smp server: notices about server pages and port 443, enable store log by default (#1501)
* smp server: notices about server pages and port 443, enable store log by default

* enable store log by default for ntf and xftp servers, fix tests
2025-03-31 16:41:11 +01:00
Evgeny
56bec06856 smp protocol: remove creating notifications from NEW command, new ideas for notifications (#1500) 2025-03-30 19:56:16 +01:00
Evgeny
04cbed90fb agent: set/update and delete contact short link data (#1499)
* agent: set/update and delete contact short link data

* delete contact link data, tests

* comments

* type signature for GHC 8.10.7
2025-03-30 11:17:25 +01:00
Evgeny
c1a6647f19 agent: join connection when 1-time invitation short link is already secured (#1496)
* agent: join connection when 1-time invitation short link is already secured

* do not pass short link to join

* delete short link record after connection
2025-03-29 09:42:56 +00:00
Evgeny Poberezkin
1dd677eec2 Merge branch 'master' into short-links 2025-03-28 19:50:18 +00:00
Evgeny
7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
sh
79adb83782 ci: add reproduce builds workflow (#1497)
* ci: add reproduce builds workflow

* ci: trigger webhook and build every day

* ci: change secrets
2025-03-27 16:06:32 +00:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny Poberezkin
0c3b25706a Merge branch 'stable' 2025-03-23 18:18:07 +00:00
sh
c3f57beafd scripts: fix script for reproducible builds (#1492)
* scripts: fix script for reproducible builds

* scripts: add automatic hash comparasion + compute file with checksums

* scripts: silence cleanup

* ci/scripts: enable parallel build (#1493)
2025-03-23 18:17:27 +00:00
Evgeny Poberezkin
aace3fd2fb 6.3.1.0 v6.3.1 2025-03-22 13:51:55 +00:00
sh
2e67ed9c4c ci/scripts: refactor reproducible builds (#1490)
* ci: build cache, remove 20.04, minor refactor

* scripts: update build reproducibility script

* script: add postgresql reproducibility

* script: no cache when building image

* scripts: force symlinks

* scripts: also download prebuilt postgresql binary

* scripts: add missing app var
2025-03-22 13:27:11 +00:00
Evgeny
614fa2b163 smp server: reduce queue expiration/idle intervals, skip expiring very old queues (#1488)
* smp server: reduce idle queue interval and queue expiration interval

* only expire recent queues (not very old)

* fix

* version
2025-03-20 08:57:47 +00:00
Evgeny
903e96bdfa rfc: smp queue protocol changes (#1485)
* rfc: smp queue protocol changes

* questions

* update

* update

* correction
2025-03-19 10:47:57 +00:00
Evgeny
5c0adcbbff Merge pull request #1467 from simplex-chat/postgres
SMP server PostgreSQL storage - feature branch (merge, dont squash)
2025-03-18 16:48:25 +00:00
sh
b2f16eeff4 ci: seperate builds (pg/standard) (#1486) 2025-03-18 09:42:53 +00:00
Evgeny
6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00
Evgeny
f4b55bfc0c smp server: CLI option to compact store log with PostgreSQL database (#1484)
* smp server: CLI option to compact store log with PostgreSQL database

* version

* fix test
2025-03-16 18:47:02 +00:00
Evgeny
fe64d42db1 smp server: split postgres support to a separate executable, to not require postgres library in the main binary (#1482)
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary

* comments

* enable server_postgres flag by default, add CPP option to test

* refactor

* change default for server_postgres to False

* diff
2025-03-16 11:37:25 +00:00
Evgeny
1b5a9f3b0c smp server: do not cache all queues from database while processing expirations (#1483)
* smp server: expire only active queues

* version

* do not cache all queues while processing expirations

* refactor

* foldWithOptions_

* version

* use shared lock when expiring all queues

* use TMVar

* comment

* rename

* remove fold options

* do not create locks in the Map for temporarily loaded queues

* fix

* revert version
2025-03-15 20:11:30 +00:00
Evgeny
fdf8bd7ee2 smp server: mask database store operations from async exception for state consistency (#1481)
* smp server: mask database store operations from async exception for state consistency

* remove unused, comments
2025-03-13 13:57:46 +00:00
Evgeny
019a32a623 smp server: optionally append store log with postgres storage (without loading and compacting, for debugging during migration) (#1480)
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)

* refactor

* remove comment
2025-03-13 10:25:56 +00:00
Evgeny
d44f09d111 tests: pass AStoreType to agent test as a parameter (#1479)
* tests: run agent tests with PostgreSQL SMP servers

* agent tests with postgres database

* enable tests

* fix store log tests

* fix test
2025-03-12 21:47:44 +00:00
Evgeny
6b5de2c51b smp server: exception on invalid migration, check queue storage settings on server start (#1478)
* smp server: check queue storage settings on server start

* fix incorrect postgres migration not throwing exception
2025-03-10 17:28:26 +00:00
shum
9410fb6f16 docker: add libpq dep 2025-03-10 10:03:44 +00:00
Evgeny
7b42aaa132 smp server: expire messages in postgres database, mark queues as deleted, combine tables (#1471)
* smp server: expire messages in postgres database

* tty

* fail if nothing updated in db

* remove old deleted queues

* index

* fix tests
2025-03-10 09:31:50 +00:00
Evgeny Poberezkin
e4b9aa9746 Merge branch 'master' into postgres 2025-03-09 21:58:39 +00:00
Evgeny Poberezkin
aa26a55df4 Merge branch 'master' into stable 2025-03-09 21:53:03 +00:00
sh
6e505f5c0b scripts: add script to reproduce binaries locally (#1477) 2025-03-08 20:51:20 +00:00
Evgeny Poberezkin
a491a1d878 6.3.0.8 v6.3.0 2025-03-07 14:30:00 +00:00
sh
36f5539b9a ci: introduce reproducible builds (#1476)
* ci: introduce reproducible builds

* ci: return 20.04

* smp server: increase timing test threshold

* ci: test outside docker

* ci: fix test step

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-03-07 14:15:18 +00:00
Evgeny
1a2afe8bfd agent: fix JSON encoding for protocol errors to be compatible with iOS (#1475) 2025-03-06 08:02:27 +00:00
Evgeny Poberezkin
9fece9ce3d 6.3.0.7 v6.3.0-beta.7 2025-03-02 22:45:07 +00:00
Evgeny Poberezkin
2e2ede5968 Merge branch 'master' into postgres 2025-03-02 22:35:28 +00:00
Evgeny
205d4ead1c smp server: remove store log backups when server starts (#1472) 2025-02-27 07:39:05 +00:00
Evgeny
80a070a8ea smp server: update STM cache only after PostgreSQL update (#1470)
* refactor journal store

* update postgres store methods to update STM cache only after DB update

* todos

* drop

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-26 11:43:08 +00:00
Evgeny
172540984c agent, smp server: PostgreSQL connection pool (#1468)
* agent, smp server: PostgreSQL connection pool

* simplify, create all connections before start

* rename

* remove imports
2025-02-25 15:33:54 +00:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny Poberezkin
f9d7b1eebc 6.3.0.6 v6.3.0-beta.6 2025-02-23 19:42:16 +00:00
Evgeny
ffbc733d58 smp server: remove duplicate progress log (#1466) 2025-02-23 19:40:49 +00:00
Evgeny
2286726d72 smp server: start options maintenance and skip-warnings (#1465)
* smp server: start options `maintenance` and `skip-warnings`

* ignore invalid parsing of the last lines

* parsingErr

* fix
2025-02-22 19:26:03 +00:00
spaced4ndy
1b8110a332 xftp server: restore file status from log (#1461)
* xftp server: restore file blocking info from log

* fix parse

* rework

* update

* rename
2025-02-21 22:43:56 +00:00
sh
dad7e1b60c simplex-servers-update: download scripts from tag (#1457)
* simplex-servers-update: download scripts from tag

* set remote version when re-executing the update script

* safeguard variables

* additional checks
2025-02-20 09:26:03 +00:00
Evgeny
72c2ddcf57 agent: move migrations (#1459) 2025-02-18 23:39:29 +00:00
Evgeny
a75e138965 smp server: remove empty queues journals when expiring messages of idle queues (#1458)
* smp server: remove empty queues journals when expiring messages of idle queues

* remove unnecessary update

* ci: update action

* rename
2025-02-18 20:04:58 +00:00
Evgeny
fa319d798a smp server: remove empty journals when opening message queue (#1456)
* smp server: remove empty journals when opening message queue

* update, do not backup state

* test

* version

* do not close queue state when queue is opened for writing

* comment

* quota = 4

* refactor openMsgQueue to prevent extra state backups

* use interval in config

* version, expire backups after 5 min

* refactor

* test
2025-02-17 23:11:34 +00:00
Evgeny Poberezkin
c192339af9 6.3.0.5 v6.3.0-beta.5 2025-02-15 15:36:44 +00:00
Evgeny
fa67d128d1 agent: fix deleting messages after delivery to avoid deleting shared message bodies (#1455)
* agent: fix deleting messages after delivery to avoid deleting shared message bodies

* fix, comments

* rename

* comment
2025-02-15 12:40:10 +00:00
Evgeny
55ff581655 SMP client: dont block on writing to sending queues (#1454)
* SMP client: dont block on writing to sending queues

* only fork if full
2025-02-15 11:31:05 +00:00