Commit Graph

22 Commits

Author SHA1 Message Date
Evgeny
3df2425162 smp server: use separate database pool for reading queues and creating service records (#1561) 2025-06-09 12:47:35 +01:00
Evgeny
5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny
07eaf9157b smp server: allow getting and deleting short links for the old contact queues (#1549)
* smp server: allow getting and deleting short links for the old contact queues

* fix verifaction of legacy contact queues

* test
2025-05-25 17:03:02 +01:00
Evgeny
850d2fa423 ntf server: PostgreSQL database storage (#1519)
* ntf server: PostgreSQL database storage

* ntf server: import/export stubs

* ntf server postgres db functions

* some notifications tests pass

* notifications tests pass

* import/export notification store logs

* fix ntf server CLI

* log in parralel

* update subscription statuses using executeMany

* fix import/export

* refactor

* fix queries

* prohibit token_id and subscription_id updates, dont update tokens and subscriptions on conflict, improve server insertion, remove duplicate tokens for import, remove subscriptions without tokens for import

* comment

* remame

* increase test delay
2025-04-25 16:47:39 +01:00
Evgeny
e4d4b51c59 smp server: fix/test database import (#1521) 2025-04-22 21:59:27 +01:00
Evgeny
b7a9542213 smp server: short links and owners for channels (#1506)
* smp server: short links and owners for channels

* types

* support mutliple rcv keys

* fix down migration, test/create server schema dump

* reduce schema dump

* parameterize type for link data by connection type

* return full connection link data

* test version

* change short link encoding

* test: print pg_dump output

* server pages, link encoding

* fix connection request when queue data and sender ID are created for old servers

* test, change pattern

* ci: install postgresql tools in runner (#1507)

* ci: install postgresql tools in runner

* ci: docker shell abort on error

* fix pattern for ghc 8.10.7

* patch ConnReqUriData SMP encoding to preserve queue mode after decoding

* test for RKEY

* fix/test store log with RKEY

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-04-10 19:09:47 +01:00
Evgeny
94ee3ceced agent: replace sndSecure with queueMode in queue URIs and records (#1502)
* agent: replace sndSecure with queueMode in queue URIs and records

* fix test
2025-03-31 16:54:29 +01:00
Evgeny
04cbed90fb agent: set/update and delete contact short link data (#1499)
* agent: set/update and delete contact short link data

* delete contact link data, tests

* comments

* type signature for GHC 8.10.7
2025-03-30 11:17:25 +01:00
Evgeny
c1a6647f19 agent: join connection when 1-time invitation short link is already secured (#1496)
* agent: join connection when 1-time invitation short link is already secured

* do not pass short link to join

* delete short link record after connection
2025-03-29 09:42:56 +00:00
Evgeny Poberezkin
1dd677eec2 Merge branch 'master' into short-links 2025-03-28 19:50:18 +00:00
Evgeny
7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny
614fa2b163 smp server: reduce queue expiration/idle intervals, skip expiring very old queues (#1488)
* smp server: reduce idle queue interval and queue expiration interval

* only expire recent queues (not very old)

* fix

* version
2025-03-20 08:57:47 +00:00
Evgeny
6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00
Evgeny
fe64d42db1 smp server: split postgres support to a separate executable, to not require postgres library in the main binary (#1482)
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary

* comments

* enable server_postgres flag by default, add CPP option to test

* refactor

* change default for server_postgres to False

* diff
2025-03-16 11:37:25 +00:00
Evgeny
1b5a9f3b0c smp server: do not cache all queues from database while processing expirations (#1483)
* smp server: expire only active queues

* version

* do not cache all queues while processing expirations

* refactor

* foldWithOptions_

* version

* use shared lock when expiring all queues

* use TMVar

* comment

* rename

* remove fold options

* do not create locks in the Map for temporarily loaded queues

* fix

* revert version
2025-03-15 20:11:30 +00:00
Evgeny
fdf8bd7ee2 smp server: mask database store operations from async exception for state consistency (#1481)
* smp server: mask database store operations from async exception for state consistency

* remove unused, comments
2025-03-13 13:57:46 +00:00
Evgeny
019a32a623 smp server: optionally append store log with postgres storage (without loading and compacting, for debugging during migration) (#1480)
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)

* refactor

* remove comment
2025-03-13 10:25:56 +00:00
Evgeny
7b42aaa132 smp server: expire messages in postgres database, mark queues as deleted, combine tables (#1471)
* smp server: expire messages in postgres database

* tty

* fail if nothing updated in db

* remove old deleted queues

* index

* fix tests
2025-03-10 09:31:50 +00:00
Evgeny
80a070a8ea smp server: update STM cache only after PostgreSQL update (#1470)
* refactor journal store

* update postgres store methods to update STM cache only after DB update

* todos

* drop

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-26 11:43:08 +00:00
Evgeny
172540984c agent, smp server: PostgreSQL connection pool (#1468)
* agent, smp server: PostgreSQL connection pool

* simplify, create all connections before start

* rename

* remove imports
2025-02-25 15:33:54 +00:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00