Commit Graph

405 Commits

Author SHA1 Message Date
Evgeny Poberezkin
d80a6b74c5 version 2025-06-12 11:10:28 +01:00
Evgeny Poberezkin
1e82104224 6.4.0.8 2025-06-09 14:20:25 +01:00
Evgeny
5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny Poberezkin
3ad8bd15a6 6.4.0.7 2025-05-29 13:14:09 +01:00
Evgeny Poberezkin
f80ed32a06 6.4.0.6 2025-05-25 17:28:39 +01:00
Evgeny
56ea2fdd56 refactor types for DB entity (#1548) 2025-05-24 18:19:11 +01:00
Evgeny Poberezkin
dae649fb87 6.4.0.5 2025-05-24 14:21:53 +01:00
Evgeny
af9ca59e51 smp server: optimize concurrency and memory usage, refactor (#1544)
* smp server: optimize concurrency and memory usage, refactor

* hide clients IntMap

* reduce STM contention

* comment

* version

* correct stats for subscriptions

* version

* comment

* remove subscribed clients from map

* version

* optimze, refactor

* version

* debug test

* enable all tests

* remove test logs

* retry failed tests with debug logging

* increase test timeout

* sync between tests
2025-05-23 12:52:18 +01:00
Evgeny Poberezkin
d352d518c2 6.4.0.4 2025-05-21 08:56:01 +01:00
Evgeny Poberezkin
f44ea0a6d8 6.4.0.3 2025-05-15 21:01:49 +01:00
Evgeny Poberezkin
e417d35cce Merge branch 'master' into ntf-storage 2025-05-14 22:20:55 +01:00
Evgeny Poberezkin
deaec3cce2 6.4.0.2 2025-05-07 07:24:06 +01:00
Evgeny
0a3d014f5d ntf server: priority db pool for notifications and client commands, concurrent resubscriptions, CLI options (#1529)
* ntf server: option to skip specific tokens when importing store log

* logs

* batch later

* subscribe in parallel, bigger queues

* ntf server: test schema and migrations

* optimize subscriptions

* log level

* resubscribe before starting

* better subscribing

* pooled concurrent resubscriptions

* 5

* async resubscribe

* typo

* priority pool

* db batch size
2025-04-30 11:40:32 +01:00
Evgeny
f024ab1c3f ntf server: prometheus metrics (#1527)
* ntf server: save prometheus stats

* info metrics

* fix test
2025-04-27 13:25:44 +01:00
Evgeny
850d2fa423 ntf server: PostgreSQL database storage (#1519)
* ntf server: PostgreSQL database storage

* ntf server: import/export stubs

* ntf server postgres db functions

* some notifications tests pass

* notifications tests pass

* import/export notification store logs

* fix ntf server CLI

* log in parralel

* update subscription statuses using executeMany

* fix import/export

* refactor

* fix queries

* prohibit token_id and subscription_id updates, dont update tokens and subscriptions on conflict, improve server insertion, remove duplicate tokens for import, remove subscriptions without tokens for import

* comment

* remame

* increase test delay
2025-04-25 16:47:39 +01:00
Evgeny Poberezkin
ec5a60430d 6.4.0.1 2025-04-22 22:01:16 +01:00
Evgeny Poberezkin
305f79d2a6 6.4.0.0 2025-04-14 10:52:05 +01:00
Evgeny Poberezkin
307a784174 Merge branch 'master' into short-links 2025-04-13 10:51:32 +01:00
Evgeny Poberezkin
9abc0fa88d 6.3.2.0 2025-04-11 23:22:48 +01:00
Evgeny
b7a9542213 smp server: short links and owners for channels (#1506)
* smp server: short links and owners for channels

* types

* support mutliple rcv keys

* fix down migration, test/create server schema dump

* reduce schema dump

* parameterize type for link data by connection type

* return full connection link data

* test version

* change short link encoding

* test: print pg_dump output

* server pages, link encoding

* fix connection request when queue data and sender ID are created for old servers

* test, change pattern

* ci: install postgresql tools in runner (#1507)

* ci: install postgresql tools in runner

* ci: docker shell abort on error

* fix pattern for ghc 8.10.7

* patch ConnReqUriData SMP encoding to preserve queue mode after decoding

* test for RKEY

* fix/test store log with RKEY

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-04-10 19:09:47 +01:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny Poberezkin
aace3fd2fb 6.3.1.0 2025-03-22 13:51:55 +00:00
Evgeny
6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00
Evgeny
f4b55bfc0c smp server: CLI option to compact store log with PostgreSQL database (#1484)
* smp server: CLI option to compact store log with PostgreSQL database

* version

* fix test
2025-03-16 18:47:02 +00:00
Evgeny
fe64d42db1 smp server: split postgres support to a separate executable, to not require postgres library in the main binary (#1482)
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary

* comments

* enable server_postgres flag by default, add CPP option to test

* refactor

* change default for server_postgres to False

* diff
2025-03-16 11:37:25 +00:00
Evgeny
1b5a9f3b0c smp server: do not cache all queues from database while processing expirations (#1483)
* smp server: expire only active queues

* version

* do not cache all queues while processing expirations

* refactor

* foldWithOptions_

* version

* use shared lock when expiring all queues

* use TMVar

* comment

* rename

* remove fold options

* do not create locks in the Map for temporarily loaded queues

* fix

* revert version
2025-03-15 20:11:30 +00:00
Evgeny Poberezkin
e4b9aa9746 Merge branch 'master' into postgres 2025-03-09 21:58:39 +00:00
Evgeny Poberezkin
a491a1d878 6.3.0.8 2025-03-07 14:30:00 +00:00
Evgeny Poberezkin
9fece9ce3d 6.3.0.7 2025-03-02 22:45:07 +00:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny Poberezkin
f9d7b1eebc 6.3.0.6 2025-02-23 19:42:16 +00:00
Evgeny
72c2ddcf57 agent: move migrations (#1459) 2025-02-18 23:39:29 +00:00
Evgeny
fa319d798a smp server: remove empty journals when opening message queue (#1456)
* smp server: remove empty journals when opening message queue

* update, do not backup state

* test

* version

* do not close queue state when queue is opened for writing

* comment

* quota = 4

* refactor openMsgQueue to prevent extra state backups

* use interval in config

* version, expire backups after 5 min

* refactor

* test
2025-02-17 23:11:34 +00:00
Evgeny Poberezkin
c192339af9 6.3.0.5 2025-02-15 15:36:44 +00:00
spaced4ndy
7ac80bffcb agent: store shared message body only once (when it is the same across messages when batching) (#1453)
* agent: store shared message body only once (when it is the same across messages when batching)

* rename

* refactor

* refactor

* save bodies and messages in single transaction

* comment

* comment

* comment

* box

* mapME

* box

* ValueOrRef

* remove instances

* refactor

* comments

* test

* refactor

* mapAccumLM compatibility with ghc 8.10.7

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 18:01:40 +00:00
spaced4ndy
0d8a1a2879 agent: encrypt messages on delivery (#1446)
* agent: save message body once (plan, schema)

* split

* new type

* bs

* encrypt on delivery

* schema

* fix test

* check pad size

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 12:35:18 +00:00
Evgeny Poberezkin
bd97cb0449 6.3.0.4 2025-02-10 15:12:54 +00:00
spaced4ndy
b633f89c1a agent: check ntf token status on registration (#1450)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* test, verify invalid

* rename

* increase delay

* disable new tests in CI

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-07 11:36:29 +00:00
Evgeny
944a22a2fb ntf server: record token invalidation reason, add date of the last token activity (#1449)
* ntf server: record token invalidation reason, add date of the last token activity

* update time

* rename

* optional

* include token ID in delivery error

* version

* protocol version

* fix, log error
2025-02-06 17:11:35 +00:00
Evgeny Poberezkin
45373e7f1f 6.3.0.3 2025-01-31 13:01:03 +00:00
Evgeny Poberezkin
2318975375 6.3.0.2 2025-01-20 16:38:14 +00:00
spaced4ndy
488c7082f3 agent: store interface (#1436) 2025-01-20 17:02:39 +04:00
spaced4ndy
fdde9863cd agent: add reopenStore function for postgres; run notification tests with postgres (#1435) 2025-01-17 16:27:37 +04:00
Evgeny Poberezkin
dadf6ec5b6 6.3.0.1 2025-01-13 08:39:41 +00:00
Evgeny Poberezkin
0a118f5631 Merge branch 'master' into postgres 2024-12-27 11:58:19 +00:00
Evgeny Poberezkin
184a95cd2a 6.3.0.0 2024-12-26 13:54:38 +00:00
spaced4ndy
b7a8df40c4 Merge branch 'master' into postgres 2024-12-25 12:15:41 +04:00
spaced4ndy
bf28902327 agent: reuse ratchet on repeat join (#1426)
* agent: reuse ratchet on repeat join

* check status

* update

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-25 00:07:45 +00:00
Evgeny Poberezkin
426bf68763 6.2.2.0 2024-12-24 21:04:58 +00:00
spaced4ndy
e557dc86e7 Merge branch 'master' into postgres 2024-12-20 17:18:54 +04:00