* smp: deliver service subscription to correct client
* tests: more resilient to concurrency
* optimize PostgreSQL query
* fix service re-association after server "downgrade"
* correctly handle service removed from server (and ID changed)
* remove unused
---------
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
* agent: remove service queue association when service ID changed
* agent: process ENDS event
* agent: send service subscription error event
* agent: test migrating to/from service subscriptions, fixes
* agent: always remove service when disabled, fix service subscriptions
* agent: use server keyhash when loading service record
* agent: process queue/service associations with delayed subscription results
* agent: service subscription events
* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)
* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers
* smp server: xor hash with triggers
* fix sql and using pgcrypto extension in tests
* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands
* agent migrations with functions/triggers
* remove agent triggers
* try tracking service subs in the agent (WIP, does not compile)
* Revert "try tracking service subs in the agent (WIP, does not compile)"
This reverts commit 59e908100d.
* comment
* agent database triggers
* service subscriptions in the client
* test / fix client services
* update schema
* fix postgres migration
* update schema
* move schema test to the end
* use static function with SQLite to avoid dynamic wrapper
* smp server: refactor message delivery to always respond SOK to subscriptions
* refactor ntf subscribe
* cancel subscription thread and reduce service subscription count when queue is deleted
* subscribe rcv service, deliver sent messages to subscribed service
* subscribe rcv service to messages (TODO delivery on subscription)
* WIP
* efficient initial delivery of messages to subscribed service
* test: delivery to client with service certificate
* test: upgrade/downgrade to/from service subscriptions
* remove service association from agent API, add per-user flag to use the service
* agent client (WIP)
* service certificates in the client
* rfc about drift detection, and SALL to mark end of message delivery
* fix test
* fix test
* add function for postgresql message storage
* update migration
* agent: support client notices
* improve
* fix, test
* rename
* cleanup
* send and process notices in more cases
* dont delete
* dont remove notice on other permanent errors
* dont remove notice if there is no notice ID in queue
* add server to error
* allow deleting
* only use notice if key hash matches
* smp server: store messages in PostgreSQL
* stored procedures to write and to expire messages
* function to export messages
* move all message functions to PostgreSQL, remove delete trigger
* comments
* import messages to db
* fix message import, add export
* fix export
* fix export
* fix compilation flags
* import messages line by line
* fix server start with database storage
* fix compilation
* comments
* smp protocol: create notification credentials via NEW command that creates the queue
* create ntf subscription for queues created with ntf credetials
* do not create ntf credentials when switching connection to another queue
* smp server: refactor subscriptions and delivery
* metric for time between MSG and ACK
* cleanup
* refactor pattern match for ghc 8.10.7
* time buckets
* split max time metric
* histogram
* fix
* protocol: refactor types and encoding
* clean
* smp server: batch commands (#1560)
* smp server: batch commands verification into one DB transaction
* ghc 8.10.7
* flatten transmission tuples
* diff
* only use batch logic if there is more than one transmission
* func
* reset NTF service when adding notifier
* version
* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"
This reverts commit 3df2425162.
* version
* Revert "version"
This reverts commit d80a6b74c5.
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)
* client certificates types (WIP)
* parameterize Transport
* protocol/schema/api changes
* agent API
* rename command
* agent subscriptions return local ClientServiceId to chat
* verify transmissions
* fix receiving client certificates, refactor
* ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions
* wait for subscriber with timeout
* safer
* refactor
* log
* remove unused
* WIP service subscriptions and associations, refactor
* process service subscriptions
* rename
* simplify switching subscriptions
* SMP service handshake with additional server handshake response
* notification delivery and STM persistence for services
* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error
* stats
* more stats
* rename SMP commands
* service subscriptions in ntf server agent (tests fail)
* fix
* refactor
* exports
* subscribe ntf server as service for associated queues
* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage
* INI option to support services
* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues
* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)
* fix test
* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription
* update RFC
* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands
* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission
* service errors, todos
* fix checkCredentials in ntf server, service errors
* refactor service auth
* refactor
* service agent: store returned queue count instead of expected
* refactor serverThread
* refactor serviceSig
* rename
* refactor, rename, test repeat NSUB service association
* respond with error to SUBS
* smp server: export/import service records between database and store log
* comment
* comments
* ghc 8.10.7
* agent: setInvitationShortLink api
* Eq instance
* allow changing link data on server, refactor
* fix
* encodings
* remove link data after connection
* Revert "encodings"
This reverts commit f8e254cca9.
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* parameterize transport by peer type (client/server)
* LogDebug level when test is retried
* support "flipped" HTTP2, fix test retry to avoid retrying pending tests
* move sync to the end of the tests
* smp server: short links and owners for channels
* types
* support mutliple rcv keys
* fix down migration, test/create server schema dump
* reduce schema dump
* parameterize type for link data by connection type
* return full connection link data
* test version
* change short link encoding
* test: print pg_dump output
* server pages, link encoding
* fix connection request when queue data and sender ID are created for old servers
* test, change pattern
* ci: install postgresql tools in runner (#1507)
* ci: install postgresql tools in runner
* ci: docker shell abort on error
* fix pattern for ghc 8.10.7
* patch ConnReqUriData SMP encoding to preserve queue mode after decoding
* test for RKEY
* fix/test store log with RKEY
---------
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
* smp server: remove locks for deleted queues, additional statistics for objects in memory
* version
* reduce queue cache usage
* less caching, refactor
* comments
* revert version
* smp protocol: short links types and other changes from RFC
* add fields for queue link ID and data
* create queue and ntf credentials with NEW command
* all tests
* simplfiy types, update rfc
* update rfc
* include SenderId in NEW request in case queue data is sent
* store queue data and generate link ID if needed
* update rfc
* agent API and types
* SMP commands and persistence for short links
* SMP client functions for short links
* agent client functions for short links
* create rcv queue with short link (TODO secret_box)
* encryption and encoding for link data, postgres client migration
* test creating short link
* get link and data, tests
* comments
* type signature
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)
* refactor
* remove comment
* smp server: queue store typeclass
* parameterize JournalMsgStore
* typeclass for queue store
* postgres WIP
* compiles, passes tests
* remove StoreType
* split migrations
* progress
* addQueueRec
* reduce type spaghetti
* remove addQueue from typeclass definition
* getQueue
* test postgres storage in SMP server
* fix schema
* comment
* import queues to postgresql
* import queues to postgresql
* log
* fix test
* counts
* ci: test smp server with postgres backend (#1463)
* ci: test smp server with postgres backend
* postgres service
* attempt
* attempt
* empty
* empty
* PGHOST attempt
* PGHOST + softlink attempt
* only softlink attempt
* working attempt (PGHOST)
* remove env var
* empty
* do not start server without DB schema, do not import when schema exists
* export database
* enable all tests, disable two tests
* option for migration confirmation
* comments
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* xftp server: use recipient ID in control port to delete and block files
* cap smp proxy agent version at 10
* version
* fix prometheus
* fix
* remove old version support
* log connection parameter on error
* tests
* log sent command tag
* log error and client version
* cap proxy version for previous destination server
* comment, test
* remove logging tag
* remove logs
* version
* SMP version 14
* version
* remove comments
* version
* servers: blocking records for content moderation
* update
* encode BLOCKED as AUTH in old versions
* update
* unblock queue command
* test, status command
* smp server: combine queue and message store into one class (WIP)
* keep deleted queue tombstones to prevent race conditions and errors when restoring
* move store log from server to store implementations
* STMQueueStore type class
* fix store closed when messages expired, handle store writing errors
* types
* version
* fix recovery from missing write journal, tests
* version
* smp server: journal store queue state validation
* validate state on update, refactor
* handle message parsing errors
* typo
* fix test, throw exception when file is smaller than expected
* fix
* fix test in ghc 8.10.7
* smp server: remove STM function from MsgStore
* polymorphic MsgStore
* jourmal storage for messages (WIP)
* more journal, test setup
* writeMsg
* test
* tryDelMsg
* delMsgQueue
* remove MsgStoreClass instance of existential wrapper for Msg stores
* store config
* extract common logic out of store instances
* add store type to config
* open journals, cache last message, tests pass
* CLI commands
* refactor import/export messages
* cli commands to import/export journal message store
* export journal without draining, import/export tests
* journal command
* import/export progress
* better progress info
* only log queue state once when importing
* logs
* handle IO errors in journal store, return as STORE error
* recover from state file errors
* fix message files after crash
* fix messages folder
