1392 Commits

Author SHA1 Message Date
Evgeny
7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny
614fa2b163 smp server: reduce queue expiration/idle intervals, skip expiring very old queues (#1488)
* smp server: reduce idle queue interval and queue expiration interval

* only expire recent queues (not very old)

* fix

* version
2025-03-20 08:57:47 +00:00
Evgeny
6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00
Evgeny
f4b55bfc0c smp server: CLI option to compact store log with PostgreSQL database (#1484)
* smp server: CLI option to compact store log with PostgreSQL database

* version

* fix test
2025-03-16 18:47:02 +00:00
Evgeny
fe64d42db1 smp server: split postgres support to a separate executable, to not require postgres library in the main binary (#1482)
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary

* comments

* enable server_postgres flag by default, add CPP option to test

* refactor

* change default for server_postgres to False

* diff
2025-03-16 11:37:25 +00:00
Evgeny
1b5a9f3b0c smp server: do not cache all queues from database while processing expirations (#1483)
* smp server: expire only active queues

* version

* do not cache all queues while processing expirations

* refactor

* foldWithOptions_

* version

* use shared lock when expiring all queues

* use TMVar

* comment

* rename

* remove fold options

* do not create locks in the Map for temporarily loaded queues

* fix

* revert version
2025-03-15 20:11:30 +00:00
Evgeny
fdf8bd7ee2 smp server: mask database store operations from async exception for state consistency (#1481)
* smp server: mask database store operations from async exception for state consistency

* remove unused, comments
2025-03-13 13:57:46 +00:00
Evgeny
019a32a623 smp server: optionally append store log with postgres storage (without loading and compacting, for debugging during migration) (#1480)
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)

* refactor

* remove comment
2025-03-13 10:25:56 +00:00
Evgeny
d44f09d111 tests: pass AStoreType to agent test as a parameter (#1479)
* tests: run agent tests with PostgreSQL SMP servers

* agent tests with postgres database

* enable tests

* fix store log tests

* fix test
2025-03-12 21:47:44 +00:00
Evgeny
6b5de2c51b smp server: exception on invalid migration, check queue storage settings on server start (#1478)
* smp server: check queue storage settings on server start

* fix incorrect postgres migration not throwing exception
2025-03-10 17:28:26 +00:00
Evgeny
7b42aaa132 smp server: expire messages in postgres database, mark queues as deleted, combine tables (#1471)
* smp server: expire messages in postgres database

* tty

* fail if nothing updated in db

* remove old deleted queues

* index

* fix tests
2025-03-10 09:31:50 +00:00
Evgeny Poberezkin
e4b9aa9746 Merge branch 'master' into postgres 2025-03-09 21:58:39 +00:00
Evgeny
1a2afe8bfd agent: fix JSON encoding for protocol errors to be compatible with iOS (#1475) 2025-03-06 08:02:27 +00:00
Evgeny Poberezkin
2e2ede5968 Merge branch 'master' into postgres 2025-03-02 22:35:28 +00:00
Evgeny
205d4ead1c smp server: remove store log backups when server starts (#1472) 2025-02-27 07:39:05 +00:00
Evgeny
80a070a8ea smp server: update STM cache only after PostgreSQL update (#1470)
* refactor journal store

* update postgres store methods to update STM cache only after DB update

* todos

* drop

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-26 11:43:08 +00:00
Evgeny
172540984c agent, smp server: PostgreSQL connection pool (#1468)
* agent, smp server: PostgreSQL connection pool

* simplify, create all connections before start

* rename

* remove imports
2025-02-25 15:33:54 +00:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny
ffbc733d58 smp server: remove duplicate progress log (#1466) 2025-02-23 19:40:49 +00:00
Evgeny
2286726d72 smp server: start options maintenance and skip-warnings (#1465)
* smp server: start options `maintenance` and `skip-warnings`

* ignore invalid parsing of the last lines

* parsingErr

* fix
2025-02-22 19:26:03 +00:00
spaced4ndy
1b8110a332 xftp server: restore file status from log (#1461)
* xftp server: restore file blocking info from log

* fix parse

* rework

* update

* rename
2025-02-21 22:43:56 +00:00
Evgeny
72c2ddcf57 agent: move migrations (#1459) 2025-02-18 23:39:29 +00:00
Evgeny
a75e138965 smp server: remove empty queues journals when expiring messages of idle queues (#1458)
* smp server: remove empty queues journals when expiring messages of idle queues

* remove unnecessary update

* ci: update action

* rename
2025-02-18 20:04:58 +00:00
Evgeny
fa319d798a smp server: remove empty journals when opening message queue (#1456)
* smp server: remove empty journals when opening message queue

* update, do not backup state

* test

* version

* do not close queue state when queue is opened for writing

* comment

* quota = 4

* refactor openMsgQueue to prevent extra state backups

* use interval in config

* version, expire backups after 5 min

* refactor

* test
2025-02-17 23:11:34 +00:00
Evgeny
fa67d128d1 agent: fix deleting messages after delivery to avoid deleting shared message bodies (#1455)
* agent: fix deleting messages after delivery to avoid deleting shared message bodies

* fix, comments

* rename

* comment
2025-02-15 12:40:10 +00:00
Evgeny
55ff581655 SMP client: dont block on writing to sending queues (#1454)
* SMP client: dont block on writing to sending queues

* only fork if full
2025-02-15 11:31:05 +00:00
spaced4ndy
7ac80bffcb agent: store shared message body only once (when it is the same across messages when batching) (#1453)
* agent: store shared message body only once (when it is the same across messages when batching)

* rename

* refactor

* refactor

* save bodies and messages in single transaction

* comment

* comment

* comment

* box

* mapME

* box

* ValueOrRef

* remove instances

* refactor

* comments

* test

* refactor

* mapAccumLM compatibility with ghc 8.10.7

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 18:01:40 +00:00
spaced4ndy
0d8a1a2879 agent: encrypt messages on delivery (#1446)
* agent: save message body once (plan, schema)

* split

* new type

* bs

* encrypt on delivery

* schema

* fix test

* check pad size

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 12:35:18 +00:00
Evgeny
a58d3540ad agent: refactor migrations (#1452)
* agent: refactor migrations

* fix postgres
2025-02-07 17:17:53 +00:00
Evgeny
5dbe6337ea ntf server: additional statistics, new invalid token reasons (#1451)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* ntf server: additional statistics

* swap

* version

* more stats

* test, verify invalid

* rename

* exclude test token from stats

* increase delay

* handle invalid token in retry, more reasons

* focus tests

* disable new tests in CI

* fix

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-07 12:19:11 +00:00
spaced4ndy
b633f89c1a agent: check ntf token status on registration (#1450)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* test, verify invalid

* rename

* increase delay

* disable new tests in CI

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-07 11:36:29 +00:00
Evgeny
944a22a2fb ntf server: record token invalidation reason, add date of the last token activity (#1449)
* ntf server: record token invalidation reason, add date of the last token activity

* update time

* rename

* optional

* include token ID in delivery error

* version

* protocol version

* fix, log error
2025-02-06 17:11:35 +00:00
Evgeny
ce24f83b64 refactor STM queues (#1447) 2025-02-05 12:04:27 +00:00
Evgeny
ae41717b9b smp server: use origin client version when processing proxied command, fixes old client sending to new server (#1443)
* smp server: pass origin client version when processing proxied command, fixes old client sending to new server

* version

* version
2025-01-28 22:04:46 +00:00
Evgeny
e78ab60c97 build: fix postgres (#1444) 2025-01-28 22:02:41 +00:00
Evgeny
b3c8358a43 agent: combine connection deletion events (#1442) 2025-01-25 08:52:40 +00:00
Evgeny
eda9e36c82 agent: track queries (#1439) 2025-01-24 10:31:50 +00:00
Evgeny
298666380f servers: handle accept error ECONNABORTED (#1437) 2025-01-20 16:32:42 +00:00
Evgeny
6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
spaced4ndy
488c7082f3 agent: store interface (#1436) 2025-01-20 17:02:39 +04:00
spaced4ndy
fdde9863cd agent: add reopenStore function for postgres; run notification tests with postgres (#1435) 2025-01-17 16:27:37 +04:00
Evgeny Poberezkin
9404a3ab63 xftp server: block stats 2025-01-14 10:46:27 +00:00
Evgeny
c528ea4f31 smp server: update versions (#1433) 2025-01-12 22:11:18 +00:00
Evgeny
3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
spaced4ndy
9d9ec8cd0b agent: remove unused functions (#1432)
* agent: remove unused functions

* cleanup imports
2025-01-10 07:59:17 +00:00
Evgeny
992b42e922 agent: option to enable/disable vacuum after SQLite migration (#1429) 2024-12-28 21:06:42 +00:00
Evgeny Poberezkin
cfde5932fd agent: restore methods for backwards compatibility with simplex-chat 2024-12-28 10:33:22 +00:00
spaced4ndy
0b986c1158 postgres: export fromOnlyBI 2024-12-26 20:13:27 +04:00
spaced4ndy
b7a8df40c4 Merge branch 'master' into postgres 2024-12-25 12:15:41 +04:00