* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)
* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers
* smp server: xor hash with triggers
* fix sql and using pgcrypto extension in tests
* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands
* agent migrations with functions/triggers
* remove agent triggers
* try tracking service subs in the agent (WIP, does not compile)
* Revert "try tracking service subs in the agent (WIP, does not compile)"
This reverts commit 59e908100d.
* comment
* agent database triggers
* service subscriptions in the client
* test / fix client services
* update schema
* fix postgres migration
* update schema
* move schema test to the end
* use static function with SQLite to avoid dynamic wrapper
* smp server: refactor message delivery to always respond SOK to subscriptions
* refactor ntf subscribe
* cancel subscription thread and reduce service subscription count when queue is deleted
* subscribe rcv service, deliver sent messages to subscribed service
* subscribe rcv service to messages (TODO delivery on subscription)
* WIP
* efficient initial delivery of messages to subscribed service
* test: delivery to client with service certificate
* test: upgrade/downgrade to/from service subscriptions
* remove service association from agent API, add per-user flag to use the service
* agent client (WIP)
* service certificates in the client
* rfc about drift detection, and SALL to mark end of message delivery
* fix test
* fix test
* add function for postgresql message storage
* update migration
* agent: support client notices
* improve
* fix, test
* rename
* cleanup
* send and process notices in more cases
* dont delete
* dont remove notice on other permanent errors
* dont remove notice if there is no notice ID in queue
* add server to error
* allow deleting
* only use notice if key hash matches
* agent: subscribe all connections
* query, version
* BoolInt
* add query to errors
* Revert "add query to errors"
This reverts commit 32a1f7fe11.
* fix optional field
* version
* limit number of in-flight subscriptions to 35000
* smp server: store messages in PostgreSQL
* stored procedures to write and to expire messages
* function to export messages
* move all message functions to PostgreSQL, remove delete trigger
* comments
* import messages to db
* fix message import, add export
* fix export
* fix export
* fix compilation flags
* import messages line by line
* fix server start with database storage
* fix compilation
* comments
* smp protocol: create notification credentials via NEW command that creates the queue
* create ntf subscription for queues created with ntf credetials
* do not create ntf credentials when switching connection to another queue
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)
* client certificates types (WIP)
* parameterize Transport
* protocol/schema/api changes
* agent API
* rename command
* agent subscriptions return local ClientServiceId to chat
* verify transmissions
* fix receiving client certificates, refactor
* ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions
* wait for subscriber with timeout
* safer
* refactor
* log
* remove unused
* WIP service subscriptions and associations, refactor
* process service subscriptions
* rename
* simplify switching subscriptions
* SMP service handshake with additional server handshake response
* notification delivery and STM persistence for services
* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error
* stats
* more stats
* rename SMP commands
* service subscriptions in ntf server agent (tests fail)
* fix
* refactor
* exports
* subscribe ntf server as service for associated queues
* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage
* INI option to support services
* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues
* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)
* fix test
* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription
* update RFC
* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands
* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission
* service errors, todos
* fix checkCredentials in ntf server, service errors
* refactor service auth
* refactor
* service agent: store returned queue count instead of expected
* refactor serverThread
* refactor serviceSig
* rename
* refactor, rename, test repeat NSUB service association
* respond with error to SUBS
* smp server: export/import service records between database and store log
* comment
* comments
* ghc 8.10.7
* agent: handle cases when last message ts is not set for notifications; set last ts for "stale" notifications when messages expired and queue is empty, to prevent repeated processing
* only log errors if they exist
* only set last ts for queue that delivered notification
* agent: join connection when 1-time invitation short link is already secured
* do not pass short link to join
* delete short link record after connection
* smp protocol: short links types and other changes from RFC
* add fields for queue link ID and data
* create queue and ntf credentials with NEW command
* all tests
* simplfiy types, update rfc
* update rfc
* include SenderId in NEW request in case queue data is sent
* store queue data and generate link ID if needed
* update rfc
* agent API and types
* SMP commands and persistence for short links
* SMP client functions for short links
* agent client functions for short links
* create rcv queue with short link (TODO secret_box)
* encryption and encoding for link data, postgres client migration
* test creating short link
* get link and data, tests
* comments
* type signature
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary
* comments
* enable server_postgres flag by default, add CPP option to test
* refactor
* change default for server_postgres to False
* diff
* smp server: queue store typeclass
* parameterize JournalMsgStore
* typeclass for queue store
* postgres WIP
* compiles, passes tests
* remove StoreType
* split migrations
* progress
* addQueueRec
* reduce type spaghetti
* remove addQueue from typeclass definition
* getQueue
* test postgres storage in SMP server
* fix schema
* comment
* import queues to postgresql
* import queues to postgresql
* log
* fix test
* counts
* ci: test smp server with postgres backend (#1463)
* ci: test smp server with postgres backend
* postgres service
* attempt
* attempt
* empty
* empty
* PGHOST attempt
* PGHOST + softlink attempt
* only softlink attempt
* working attempt (PGHOST)
* remove env var
* empty
* do not start server without DB schema, do not import when schema exists
* export database
* enable all tests, disable two tests
* option for migration confirmation
* comments
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* agent: store shared message body only once (when it is the same across messages when batching)
* rename
* refactor
* refactor
* save bodies and messages in single transaction
* comment
* comment
* comment
* box
* mapME
* box
* ValueOrRef
* remove instances
* refactor
* comments
* test
* refactor
* mapAccumLM compatibility with ghc 8.10.7
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* agent: save message body once (plan, schema)
* split
* new type
* bs
* encrypt on delivery
* schema
* fix test
* check pad size
* rename
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Evgeny