Commit Graph

278 Commits

Author SHA1 Message Date
Evgeny
d3754b36bc smp server: do not use queue cache with PostgreSQL message storage (#1637)
* smp server: do not use queue cache with PostgreSQL message storage

* fix loading queues via notifier IDs
2025-09-15 07:51:27 +01:00
Evgeny Poberezkin
112cd9d5f4 Merge branch 'stable' 2025-09-14 14:42:05 +01:00
Evgeny
8a7991a376 agent: option to set SQLite database backup path (#1636)
* agent: option to set SQLite database backup path

* fix test compilation
2025-09-14 12:41:49 +01:00
Evgeny
9cfdae3494 smp server: uniterruptible mask for DB operations (#1635) 2025-09-13 19:08:07 +01:00
Evgeny
9be9a76b42 smp server: improve message expiration (#1634)
* smp server: limit by time the queues to export journal messages for

* pass queue/msg thresholds separately

* reset db connection on errors

* Revert "smp server: limit by time the queues to export journal messages for"

This reverts commit d3bc0cba4b.

* fix test compilation

* flag to expire messages

* improve test

* expire messages newer than quota
2025-09-13 11:35:34 +01:00
Evgeny
6c66cf367a smp server: set message counts correctly after import (#1632) 2025-09-12 13:08:11 +01:00
Evgeny
a137d01c90 smp server: improve counts management and expiration for postgres message store (#1631)
* smp server: improve counts management and expiration for postgres message store

* one update

* fix expiration
2025-09-12 11:39:35 +01:00
Evgeny
bac6ea6e91 smp server: store messages in PostgreSQL (#1622)
* smp server: store messages in PostgreSQL

* stored procedures to write and to expire messages

* function to export messages

* move all message functions to PostgreSQL, remove delete trigger

* comments

* import messages to db

* fix message import, add export

* fix export

* fix export

* fix compilation flags

* import messages line by line

* fix server start with database storage

* fix compilation

* comments
2025-09-11 20:22:55 +01:00
Evgeny
0c1030cf02 smp server: faster export (#1626)
* smp server: faster export

* flush

* export messages with PostgreSQL database

* remove flush

* batch message writes

* Revert "batch message writes"

This reverts commit 61fb3c3b9e.

* remove $!

* fast journal export

* another approach

* Revert "smp server: remove dependency of message size on the version (#1627)"

This reverts commit 8fea15245a.

* style

* faster?

* cleanup

* cleanup

* refactor

* refactor

* concurrent read messages

* Revert "concurrent read messages"

This reverts commit 05a32e6491.

* concurrent read/write

* parameter to export to/import from another message log file

* Revert "parameter to export to/import from another message log file"

This reverts commit 4e88b035c4.

* Revert "concurrent read/write"

This reverts commit a8eab1f7e3.
2025-09-10 18:22:58 +01:00
Evgeny
4386dd8b5a smp server: option to expire messages when new message is sent (now off by default) (#1625)
* smp server: option to expire messages when new message is sent (now off by default)

* fix test
2025-09-07 09:02:32 +01:00
Evgeny
cb3250e7b4 servers: better socket leak prevention during TLS handshake, add NetworkError type to better diagnose connection errors (#1619)
* servers: better socket leak prevention during TLS handshake

* log tcp connection errors

* more detailed network error

* log full address

* rename error

* add encodings for NetworkError

* refactor

* comment

* bind

* style

* remove parameters of NETWORK error from encoding
2025-09-02 16:07:37 +01:00
Evgeny
77bf76e1f9 smp server: fix in-memory server not restoring queue/service associations after 2+ restarts (#1618)
* smp server: fix in-memory server not restoring queue/service associations after 2+ restarts

* fix test, do not reuse database
2025-09-01 17:45:31 +01:00
Evgeny
e48bedeaf2 servers: fix server pages when source code is not specified, include commit SHA in printed version and in web page. (#1608)
* smp server: fix server pages when source code is not specified

* servers: include git commit in version

* flexible alpn

* fix test

* fix ghc 8.10.7 build
2025-08-23 19:59:00 +01:00
Evgeny
96e8b4a146 smp server: wrap all queries in transactions (#1603)
* smp server: wrap all queries in transactions

* fix test

* fix schema test
2025-08-17 10:38:20 +01:00
Evgeny
86fb2cddc5 smp server: support short SimpleX addresses in server information / pages (#1600) 2025-08-12 07:31:32 +01:00
Evgeny
f4e7469f96 xftp server: prometheus metrics (#1595)
* xftp server: prometheus metrics

* update

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-07-30 16:18:49 +01:00
Evgeny
c0f357d817 servers: control port session improvements (#1591)
* servers: prohibit changing role during control port session

* quota for blocked queues

* allow disabling blocking and quota

* fix test

* fix INI file
2025-07-21 15:19:43 +01:00
Evgeny
40fc09a93d smp server: additional stat counter for ntf credentials created together with the queue (#1589)
* smp server: additional stat counter for ntf credentials created together with the queue

* fix prometheus

* fix test

* fix qSub
2025-07-18 09:26:25 +01:00
Evgeny
62733ef4c1 smp server: refactor subscriptions and delivery in order to always response SOK on subscription with an optional message to follow. (#1573)
* smp server: refactor subscriptions and delivery

* metric for time between MSG and ACK

* cleanup

* refactor pattern match for ghc 8.10.7

* time buckets

* split max time metric

* histogram

* fix
2025-07-12 14:18:38 +01:00
Evgeny
ba3c75e58c smp server: correctly track if ntf service is subscribed and total subscribed queues count (fixes race condition between NSUB and NSUBS from notification server) (#1583) 2025-07-07 18:49:47 +01:00
spaced4ndy
1b8613d767 build: do not exclude postgres modules for client_library flag (#1579)
* don't exclude postgres modules for client_library flag

* fix build with client and server postgres flag

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-07-04 17:31:57 +01:00
Evgeny
455360205c smp server: prometheus histogram for message confirmation times (ACK) (#1575)
* time buckets

* split max time metric

* histogram

* histogram for confirmed delivery times

* gaugehistogram

* fix created, _ in gauge_histogram

* remove comments

* fix metrics
2025-06-20 13:46:59 +01:00
Evgeny
b47d28a22a smp server: prometheus metrics for delievered and not acknowledged messages (to monitor stuck deliveries) (#1572) 2025-06-17 21:10:09 +01:00
Evgeny
da37384335 smp server: batch commands (#1559)
* protocol: refactor types and encoding

* clean

* smp server: batch commands (#1560)

* smp server: batch commands verification into one DB transaction

* ghc 8.10.7

* flatten transmission tuples

* diff

* only use batch logic if there is more than one transmission

* func

* reset NTF service when adding notifier

* version

* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"

This reverts commit 3df2425162.

* version

* Revert "version"

This reverts commit d80a6b74c5.
2025-06-12 23:05:04 +01:00
Evgeny
1658048c2c Revert "smp server: use separate database pool for reading queues and creating service records (#1561)" (#1564)
This reverts commit 3df2425162.
2025-06-12 11:04:43 +01:00
Evgeny
cf8088ac6a smp server: do not require ALPN for HTTP to fix Android asset links, do not use port 443 in server-server SMP connections (#1562)
* log alpn

* always use HTTP when SNI is sent, regardless of ALPN

* decide credential based on SNI

* do not use web port in SMP/Ntf servers connecting to SMP servers

* simpler

* refactor

* fix
2025-06-10 22:09:07 +01:00
Evgeny
3df2425162 smp server: use separate database pool for reading queues and creating service records (#1561) 2025-06-09 12:47:35 +01:00
Evgeny
5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny
8e86c97a13 servers: include supported ALPNs in server transport config (#1557) 2025-06-03 13:35:35 +01:00
Evgeny
56851365b1 servers: parameterize SMP agent to optimize subscriptions (#1555)
* servers: parameterize SMP agent to optimize subscriptions

* refactor ClientCommand to avoid additional map in the notification server agent

* rename
2025-06-01 16:14:52 +01:00
Evgeny
a9814bb6d3 smp server: parameterize server by store type to avoid boxing/unboxing client handles (#1554)
* smp server: parameterize server by store type to avoid boxing/unboxing client handles

* ghc 8.10.7
2025-05-29 22:17:28 +01:00
Evgeny
07eaf9157b smp server: allow getting and deleting short links for the old contact queues (#1549)
* smp server: allow getting and deleting short links for the old contact queues

* fix verifaction of legacy contact queues

* test
2025-05-25 17:03:02 +01:00
Evgeny
ffecd4a17a parameterize transport by peer type (client/server) (#1545)
* parameterize transport by peer type (client/server)

* LogDebug level when test is retried

* support "flipped" HTTP2, fix test retry to avoid retrying pending tests

* move sync to the end of the tests
2025-05-24 14:34:22 +01:00
Evgeny
af9ca59e51 smp server: optimize concurrency and memory usage, refactor (#1544)
* smp server: optimize concurrency and memory usage, refactor

* hide clients IntMap

* reduce STM contention

* comment

* version

* correct stats for subscriptions

* version

* comment

* remove subscribed clients from map

* version

* optimze, refactor

* version

* debug test

* enable all tests

* remove test logs

* retry failed tests with debug logging

* increase test timeout

* sync between tests
2025-05-23 12:52:18 +01:00
Evgeny
0a3d014f5d ntf server: priority db pool for notifications and client commands, concurrent resubscriptions, CLI options (#1529)
* ntf server: option to skip specific tokens when importing store log

* logs

* batch later

* subscribe in parallel, bigger queues

* ntf server: test schema and migrations

* optimize subscriptions

* log level

* resubscribe before starting

* better subscribing

* pooled concurrent resubscriptions

* 5

* async resubscribe

* typo

* priority pool

* db batch size
2025-04-30 11:40:32 +01:00
Evgeny
f024ab1c3f ntf server: prometheus metrics (#1527)
* ntf server: save prometheus stats

* info metrics

* fix test
2025-04-27 13:25:44 +01:00
Evgeny
850d2fa423 ntf server: PostgreSQL database storage (#1519)
* ntf server: PostgreSQL database storage

* ntf server: import/export stubs

* ntf server postgres db functions

* some notifications tests pass

* notifications tests pass

* import/export notification store logs

* fix ntf server CLI

* log in parralel

* update subscription statuses using executeMany

* fix import/export

* refactor

* fix queries

* prohibit token_id and subscription_id updates, dont update tokens and subscriptions on conflict, improve server insertion, remove duplicate tokens for import, remove subscriptions without tokens for import

* comment

* remame

* increase test delay
2025-04-25 16:47:39 +01:00
Evgeny
e4d4b51c59 smp server: fix/test database import (#1521) 2025-04-22 21:59:27 +01:00
Evgeny
b7a9542213 smp server: short links and owners for channels (#1506)
* smp server: short links and owners for channels

* types

* support mutliple rcv keys

* fix down migration, test/create server schema dump

* reduce schema dump

* parameterize type for link data by connection type

* return full connection link data

* test version

* change short link encoding

* test: print pg_dump output

* server pages, link encoding

* fix connection request when queue data and sender ID are created for old servers

* test, change pattern

* ci: install postgresql tools in runner (#1507)

* ci: install postgresql tools in runner

* ci: docker shell abort on error

* fix pattern for ghc 8.10.7

* patch ConnReqUriData SMP encoding to preserve queue mode after decoding

* test for RKEY

* fix/test store log with RKEY

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-04-10 19:09:47 +01:00
Evgeny
a2a4b80af4 agent: padded encryption for link data, shorten/restore short links, tests (#1505)
* agent: padded encryption for link data, tests

* lambda

* test short links via proxy

* tests: server persistence with short links

* rfc: group links

* shorten, restore, test short links encoding

* rfc
2025-04-05 07:45:39 +01:00
Evgeny
94ee3ceced agent: replace sndSecure with queueMode in queue URIs and records (#1502)
* agent: replace sndSecure with queueMode in queue URIs and records

* fix test
2025-03-31 16:54:29 +01:00
Evgeny Poberezkin
dd67de4d71 Merge branch 'master' into short-links 2025-03-31 16:54:07 +01:00
Evgeny
285fd93c32 smp server: notices about server pages and port 443, enable store log by default (#1501)
* smp server: notices about server pages and port 443, enable store log by default

* enable store log by default for ntf and xftp servers, fix tests
2025-03-31 16:41:11 +01:00
Evgeny
04cbed90fb agent: set/update and delete contact short link data (#1499)
* agent: set/update and delete contact short link data

* delete contact link data, tests

* comments

* type signature for GHC 8.10.7
2025-03-30 11:17:25 +01:00
Evgeny
c1a6647f19 agent: join connection when 1-time invitation short link is already secured (#1496)
* agent: join connection when 1-time invitation short link is already secured

* do not pass short link to join

* delete short link record after connection
2025-03-29 09:42:56 +00:00
Evgeny Poberezkin
1dd677eec2 Merge branch 'master' into short-links 2025-03-28 19:50:18 +00:00
Evgeny
7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny
614fa2b163 smp server: reduce queue expiration/idle intervals, skip expiring very old queues (#1488)
* smp server: reduce idle queue interval and queue expiration interval

* only expire recent queues (not very old)

* fix

* version
2025-03-20 08:57:47 +00:00
Evgeny
6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00