* smp server: refactor subscriptions and delivery
* metric for time between MSG and ACK
* cleanup
* refactor pattern match for ghc 8.10.7
* time buckets
* split max time metric
* histogram
* fix
* protocol: refactor types and encoding
* clean
* smp server: batch commands (#1560)
* smp server: batch commands verification into one DB transaction
* ghc 8.10.7
* flatten transmission tuples
* diff
* only use batch logic if there is more than one transmission
* func
* reset NTF service when adding notifier
* version
* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"
This reverts commit 3df2425162.
* version
* Revert "version"
This reverts commit d80a6b74c5.
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)
* client certificates types (WIP)
* parameterize Transport
* protocol/schema/api changes
* agent API
* rename command
* agent subscriptions return local ClientServiceId to chat
* verify transmissions
* fix receiving client certificates, refactor
* ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions
* wait for subscriber with timeout
* safer
* refactor
* log
* remove unused
* WIP service subscriptions and associations, refactor
* process service subscriptions
* rename
* simplify switching subscriptions
* SMP service handshake with additional server handshake response
* notification delivery and STM persistence for services
* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error
* stats
* more stats
* rename SMP commands
* service subscriptions in ntf server agent (tests fail)
* fix
* refactor
* exports
* subscribe ntf server as service for associated queues
* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage
* INI option to support services
* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues
* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)
* fix test
* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription
* update RFC
* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands
* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission
* service errors, todos
* fix checkCredentials in ntf server, service errors
* refactor service auth
* refactor
* service agent: store returned queue count instead of expected
* refactor serverThread
* refactor serviceSig
* rename
* refactor, rename, test repeat NSUB service association
* respond with error to SUBS
* smp server: export/import service records between database and store log
* comment
* comments
* ghc 8.10.7
* smp server: short links and owners for channels
* types
* support mutliple rcv keys
* fix down migration, test/create server schema dump
* reduce schema dump
* parameterize type for link data by connection type
* return full connection link data
* test version
* change short link encoding
* test: print pg_dump output
* server pages, link encoding
* fix connection request when queue data and sender ID are created for old servers
* test, change pattern
* ci: install postgresql tools in runner (#1507)
* ci: install postgresql tools in runner
* ci: docker shell abort on error
* fix pattern for ghc 8.10.7
* patch ConnReqUriData SMP encoding to preserve queue mode after decoding
* test for RKEY
* fix/test store log with RKEY
---------
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
* agent: padded encryption for link data, tests
* lambda
* test short links via proxy
* tests: server persistence with short links
* rfc: group links
* shorten, restore, test short links encoding
* rfc
* smp server: remove locks for deleted queues, additional statistics for objects in memory
* version
* reduce queue cache usage
* less caching, refactor
* comments
* revert version
* smp protocol: short links types and other changes from RFC
* add fields for queue link ID and data
* create queue and ntf credentials with NEW command
* all tests
* simplfiy types, update rfc
* update rfc
* include SenderId in NEW request in case queue data is sent
* store queue data and generate link ID if needed
* update rfc
* agent API and types
* SMP commands and persistence for short links
* SMP client functions for short links
* agent client functions for short links
* create rcv queue with short link (TODO secret_box)
* encryption and encoding for link data, postgres client migration
* test creating short link
* get link and data, tests
* comments
* type signature
* smp server: expire only active queues
* version
* do not cache all queues while processing expirations
* refactor
* foldWithOptions_
* version
* use shared lock when expiring all queues
* use TMVar
* comment
* rename
* remove fold options
* do not create locks in the Map for temporarily loaded queues
* fix
* revert version
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)
* refactor
* remove comment
* smp server: queue store typeclass
* parameterize JournalMsgStore
* typeclass for queue store
* postgres WIP
* compiles, passes tests
* remove StoreType
* split migrations
* progress
* addQueueRec
* reduce type spaghetti
* remove addQueue from typeclass definition
* getQueue
* test postgres storage in SMP server
* fix schema
* comment
* import queues to postgresql
* import queues to postgresql
* log
* fix test
* counts
* ci: test smp server with postgres backend (#1463)
* ci: test smp server with postgres backend
* postgres service
* attempt
* attempt
* empty
* empty
* PGHOST attempt
* PGHOST + softlink attempt
* only softlink attempt
* working attempt (PGHOST)
* remove env var
* empty
* do not start server without DB schema, do not import when schema exists
* export database
* enable all tests, disable two tests
* option for migration confirmation
* comments
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* smp server: remove empty journals when opening message queue
* update, do not backup state
* test
* version
* do not close queue state when queue is opened for writing
* comment
* quota = 4
* refactor openMsgQueue to prevent extra state backups
* use interval in config
* version, expire backups after 5 min
* refactor
* test
* xftp server: use recipient ID in control port to delete and block files
* cap smp proxy agent version at 10
* version
* fix prometheus
* fix
* remove old version support
* log connection parameter on error
* tests
* log sent command tag
* log error and client version
* cap proxy version for previous destination server
* comment, test
* remove logging tag
* remove logs
* version
* SMP version 14
* version
* remove comments
* version
* servers: blocking records for content moderation
* update
* encode BLOCKED as AUTH in old versions
* update
* unblock queue command
* test, status command
* smp server: combine queue and message store into one class (WIP)
* keep deleted queue tombstones to prevent race conditions and errors when restoring
* move store log from server to store implementations
* STMQueueStore type class
* fix store closed when messages expired, handle store writing errors
* types
* version
* fix recovery from missing write journal, tests
* version
* smp server: journal store queue state validation
* validate state on update, refactor
* handle message parsing errors
* typo
* fix test, throw exception when file is smaller than expected
* fix
* fix test in ghc 8.10.7
* core: updated journal store API
* parameterize JournalState by read/write type
* collect stats when importing/expiring
* compare stored stats and store for messages and notifications
* simplify
* smp server: remove STM function from MsgStore
* polymorphic MsgStore
* jourmal storage for messages (WIP)
* more journal, test setup
* writeMsg
* test
* tryDelMsg
* delMsgQueue
* remove MsgStoreClass instance of existential wrapper for Msg stores
* store config
* extract common logic out of store instances
* add store type to config
* open journals, cache last message, tests pass
* CLI commands
* refactor import/export messages
* cli commands to import/export journal message store
* export journal without draining, import/export tests
* journal command
* import/export progress
* better progress info
* only log queue state once when importing
* logs
* handle IO errors in journal store, return as STORE error
* recover from state file errors
* fix message files after crash
* fix messages folder
* .401
* stats for undelivered notifications
* logs, stats
* control port show ntf client IDs
* check that Ntf client is still current and that queue is not full, drop notifications otherwise
* prevent losing notifications when client is not current or queue full
* add log when no notifications, remove some logs
* reduce STM transaction
* revert version change
