Commit Graph

297 Commits

Author SHA1 Message Date
Evgeny
2ea9a9a143 agent: finalize initial service subscriptions, remove associations on service ID changes (#1672)
* agent: remove service/queue associations when service ID changes

* agent: check that service ID in NEW response matches session ID in transport session

* agent subscription WIP

* test

* comment

* enable tests

* update queries

* agent: option to add SQLite aggregates to DB connection  (#1673)

* agent: add build_relations_vector function to sqlite

* update aggregate

* use static aggregate

* remove relations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add test, treat BAD_SERVICE as temp error, only remove queue associations on service errors

* add packZipWith for backward compatibility with GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-12-05 20:46:48 +00:00
Evgeny Poberezkin
ff7bdbcbeb Merge branch 'master' into rcv-services 2025-12-03 17:23:59 +00:00
Evgeny
cf9b7e5b6a agent: option to add SQLite functions to DB connection (#1674)
* agent: option to add SQLite functions to DB connection

* add module
2025-12-03 16:10:35 +00:00
Evgeny
3ccf854865 servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1668)
* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)

* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers

* smp server: xor hash with triggers

* fix sql and using pgcrypto extension in tests

* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands

* agent migrations with functions/triggers

* remove agent triggers

* try tracking service subs in the agent (WIP, does not compile)

* Revert "try tracking service subs in the agent (WIP, does not compile)"

This reverts commit 59e908100d.

* comment

* agent database triggers

* service subscriptions in the client

* test / fix client services

* update schema

* fix postgres migration

* update schema

* move schema test to the end

* use static function with SQLite to avoid dynamic wrapper
2025-11-25 16:55:59 +00:00
Evgeny
99f40ae109 support for additional database migrations (#1644) 2025-09-27 11:21:38 +01:00
Evgeny
8a7991a376 agent: option to set SQLite database backup path (#1636)
* agent: option to set SQLite database backup path

* fix test compilation
2025-09-14 12:41:49 +01:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny
72c2ddcf57 agent: move migrations (#1459) 2025-02-18 23:39:29 +00:00
Evgeny
a58d3540ad agent: refactor migrations (#1452)
* agent: refactor migrations

* fix postgres
2025-02-07 17:17:53 +00:00
Evgeny
eda9e36c82 agent: track queries (#1439) 2025-01-24 10:31:50 +00:00
spaced4ndy
488c7082f3 agent: store interface (#1436) 2025-01-20 17:02:39 +04:00
spaced4ndy
fdde9863cd agent: add reopenStore function for postgres; run notification tests with postgres (#1435) 2025-01-17 16:27:37 +04:00
Evgeny
992b42e922 agent: option to enable/disable vacuum after SQLite migration (#1429) 2024-12-28 21:06:42 +00:00
spaced4ndy
cf66aadc20 postgres: store implementation, conditional compilation (#1421)
* postgres: implementation wip

* to from field

* agent store compiles

* methods

* create store

* tests wip

* migration tests pass

* tests compile

* fix tests

* tests wip

* bool int

* tests wip

* tests wip

* more boolint

* more fixes

* more fields pass

* more fixes

* binary

* instances, binary

* test passes

* remove todos, more tests pass

* fix conflict

* fix bool

* fix sequence breaking

* fix insertedRowId

* skip ratchet re-synchronization tests

* after test

* file tests

* after test

* rename

* remove comment

* format

* remove unused

* suppress notices

* fixes

* move

* fix

* instance

* instance2

* fix

* instances

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-20 15:54:58 +04:00
spaced4ndy
64149599de postgres: db interfaces wip (sqlite passes) (#1419) 2024-12-12 17:42:58 +04:00
spaced4ndy
9893935e7c agent: fix acceptContact inconsistent state after app restart (#1412) 2024-12-05 19:27:21 +04:00
spaced4ndy
a8471eed5b ntf: get messages for multiple last notifications (#1378)
* ntf: get messages for multiple last notifications (#1352)

* ntf: separate get ntf conns api (#1379)

* ntf: separate get ntf conns api

* nonempty

* update

* update

* remove single get api

* fix test

* refactor

* refactor

* ntf: batch get connections (#1387)

* ntf: batch get apis

* works

* fix

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-25 15:39:41 +04:00
spaced4ndy
49935473ae ntf: agent to notfiy MSGNTF on successful ack (#1355)
* ntf: agent to notfiy MSGNTF on successful ack

* move msgntf

* release lock earlier

* fix tests

* do not release get lock

* fix

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-06 19:48:22 +01:00
spaced4ndy
243548631a ntf: repeat failed token deletions (#1346) 2024-10-01 16:30:14 +04:00
spaced4ndy
9cf6c97137 ntf: batch ntf worker command processing (#1337)
* ntf: batch ntf worker command processing

* remove comment

* change batch size

* wip

* catch

* refactor

* refactor

* batch check

* refactor

* reschedule

* increase ntfSubCheckInterval

* first check interval

* check more statuses

* refactor, remove foldr'

* refactor 2

* refactor client

* refactor 3

* ntf server: improve support for batched commands (#1340)

* re-create ntf subscriptions on NTF AUTH errors

* name

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-29 18:50:38 +01:00
spaced4ndy
81fcdf8ac9 ntf: batch smp worker command processing (#1331)
* ntf: refactor smp worker (separate command processing into steps)

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-25 15:00:06 +04:00
Evgeny
1f67b403de agent: function to get multiple work items (#1330) 2024-09-25 10:55:45 +01:00
spaced4ndy
ea67b34ba7 ntf: batch supervisor commands (#1313) 2024-09-13 15:26:57 +04:00
spaced4ndy
e247f691cb agent: do not reuse notifier credentials when re-creating notification subscription; on deleting ntf sub delete directly on smp server skipping ntf server (#1311)
* agent: change notifier credentials when re-creating notification subscription

* skip ndel

* enable tests

* fix race

* Revert "fix race"

This reverts commit ed9b18e8a7.

* delete record

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-12 14:02:51 +04:00
Evgeny
dab1980d79 xftp: report receive file error with redirected file ID, when redirect is present (#1304)
* xftp: report receive file error with redirected file ID, when redirect is present

* fix test
2024-09-09 08:08:16 +01:00
Evgeny
d5efe3406a agent: fix race when sending a message to the deleted connection (#1296) 2024-09-02 15:07:16 +01:00
Evgeny
ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny
ab4acadcc1 agent: fix stuck connection commands when server is not responding (#1276)
* agent: fix stuck connection commands when server is not responding

* fix

* rename

* test SKEY retry
2024-08-24 13:54:20 +01:00
Diogo
1cbf8c0015 agent: support changing user of the new connection (#1267)
* agent: support new connection user id update

* another way for assertion

* add more tests to setConnUserId

* remove fdescribes

* allow rcv connection to change user id

* add functional test to api

* remove fdescribe

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-08-20 10:57:09 +01:00
Evgeny
571d148bdf agent: do not start delivery workers when there are no messages to deliver (#1263)
* agent: use weak ThreadId and forkIO in workers instead of async (reduce memory)

* agent: do not start and exit delivery workers when there are no messages to deliver (#1264)

* agent: exit delivery workers when no messages to deliver

* only start delivery workers when there are pending messages

* fix

* focus test

* enable all tests

* lift

* do not exit workers when there is no work
2024-08-18 21:30:06 +01:00
Evgeny
f229e135e3 agent: reduce subscription transactions (#1259)
* agent: reduce subscription transactions

* nub

* remove commented
2024-08-16 10:06:22 +01:00
Evgeny
7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny Poberezkin
3753379ae4 agent: support dababase access with high priority for more responsive user actions (#1248)
* agent: support dababase access with high priority for more responsive user actions

* rename, export

* fix
2024-07-29 18:57:16 +01:00
Evgeny Poberezkin
051bf38bc7 agent: remove delays in notification processing, add notification stats (#1235)
* agent: remove delays in notification processing, add notification stats

* do not replace token after failed check

* more stats

* refactor

* fix

* backwards compatible JSON parsing

* retry deleting subscription on temporary error

* remove attempts to get multiple notification messages

* fix JSON decoding to be backwards compatible
2024-07-22 15:42:34 +01:00
spaced4ndy
ae8e1c5e9a agent: servers stats improvements, fixes (#1208)
* agent: reset stats startedAt time in memory

* getAgentSubsSummary

* change sub counting

* ack statistics

* add import

* instance

* Revert "instance"

This reverts commit 1f63740d56.

* Revert "add import"

This reverts commit ef72df8014.

* modify sub counting

* modify conn creation counting

* use int64

* file size stats

* remove import

* ack err counting

* conn del stats

* format

* new data

* add data

* toKB

* restore connCompleted

* use Int for counts

* use rq from scope

* remove getAgentSubsSummary

* fix connCompleted

* fix

* revert disabling stats

* use srv from scope

* combine ack stats

* modify

* comment

* count subs

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-07-03 12:36:15 +01:00
Evgeny Poberezkin
9ee684b0f4 rfc: faster handshake protocol (#1203)
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
2024-06-30 08:36:24 +01:00
spaced4ndy
c788692687 agent: servers summary types, api (#1202)
* agent: servers summary types, api [wip]

* encoding

* export

* Revert "export"

This reverts commit cd9f315fe8.

* comment

* rename

* simplify types

* uncomment

* comment

* rework

* comment, exports

* save, restore stats wip

* remove

* rename

* save stats periodically

* sigint, sigterm experiments

* corrections

* remove some proxy stats

* increase stat

* proposed stats

* fields

* Revert "sigint, sigterm experiments"

This reverts commit f876fbd418.

* wip

* retries -> attempts

* errs

* fix

* other errs

* more stat tracking

* sub stats

* remove xftp successes stats

* xftp stats tracking

* revert

* revert

* refactor

* remove imports

* comment

* Revert "refactor"

This reverts commit 26c368d82a.

* Revert "revert"

This reverts commit 4c9e3753b5.

* Revert "revert"

This reverts commit 6f65644053.

* todos

* persistence

* rename, fix

* config

* comment

* add started at to summary

* delete stats on user deletion

* reset api

* move

* getAgentServersSummary collect state logic

* corrections

* corrections

* remove

* rework

* decrease contention

* update

* more stats

* count sentProxied

* count subs

* remove unused

* comment

* remove comment

* comment

* export

* refactor

* cleanup

* intervals

* refactor

* refactor2

* refactor3

* refactor4

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-25 09:42:59 +04:00
Evgeny Poberezkin
3dab330480 use throwE instead of throwError (#1187)
* use throwE instead of throwError

* test delay
2024-06-05 11:20:50 +01:00
Evgeny Poberezkin
2ff5f5a832 agent: add context to CMD error (#1167)
* agent: add context to CMD error

* tests, more warnings

* fix tests

* log TBQueue sizes

* log locks

* more logs

* log sendMessagesB

* fix test

* log length

* refactor

* remove logging

* revert lock scope change

* cleanup

* add string error to A_PROHIBITED

* remove

* remove test limitations

* language
2024-05-24 14:13:01 +01:00
spaced4ndy
2e5433676e xftp: check proxy before downloading from unknown server (#1102)
* xftp: check proxy before downloading from unknown server

* corrections

* remove import

* tests

* comment

* tests

* don't wrap into internal error

* fix tests
2024-05-20 14:14:04 +01:00
Evgeny Poberezkin
f51cf1deac agent: use MVar for DB connection for more fair connection distribution between threads (#1147) 2024-05-14 20:04:51 +01:00
Evgeny Poberezkin
4455b8bd0e agent: do not throw exception when command is created for deleted connection (#1150)
* agent: do not throw exception when command is created for deleted connection

* convert database busy/locked to critical alert
2024-05-13 08:10:40 +01:00
Evgeny Poberezkin
1219446996 dispose agent instances, fix tests, (#1089)
* dispose agent instances in tests

* fix quota test

* tests: fix tests with -threaded (#1088)

* fix some tests

* match RTS opts with apps

* less verbose rts stats

* enable sqlite extended error codes

* clean up

* unfocus

* remove extendedErrorCode

It's actually setExtendedResultCodes, which isn't yet available.

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* fix switch test

* fix

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-04-10 19:34:02 +01:00
Evgeny Poberezkin
84b8c8417b revert base64 change (#1081) 2024-04-03 09:27:36 +01:00
Alexander Bondarenko
ee90ea6a69 replace base64-bytestring with base64 (#1065)
* replace base64-bytestring with base64

* minify

* use bytestring-0.10 compatible fork

PR pending...

* bump base64 fork with text compat

* move compat details to modules

* switch repo

* add back module

* cleanup

* minify

* clean imports

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-27 23:35:09 +00:00
Evgeny Poberezkin
11288866f9 pqdr: refactor 2024-03-07 12:41:10 +00:00
Evgeny Poberezkin
4ffb6a348a pqdr: use different newtypes for supporting and enabling PQ encryption in connections (#1031)
* pqdr: use different newtypes for supporting and enabling PQ encryption in connections

* rename field, fix test

* refactor
2024-03-06 21:28:03 +00:00
Evgeny Poberezkin
b435a4dacb envelope sizes dependent on PQ encryption (#1028)
* envelope sizes dependent on PQ encryption (WIP)

* add "supported" flag to ratchets, update this flag on ratchet resync

* change connection PQ status on sendMessage

* comment, fix

* refactor
2024-03-06 16:38:30 +00:00
Evgeny Poberezkin
b050cf5027 double ratchet versioning for post-quantum encryption (#1025)
* correctly parse new Ratchet fields when omitted

* rfc: migrating connection versions to pqdr

* update rfc

* WIP (dont commit)

* rename versions

* update ratchet version based on PQ encryption feature flag

* remove duplicate function

* synchronize ratchet, fix tests, refactor

* comments

* test

* pattern
2024-03-05 17:07:15 +00:00
Evgeny Poberezkin
ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00