* agent: subscribe all connections
* query, version
* BoolInt
* add query to errors
* Revert "add query to errors"
This reverts commit 32a1f7fe11.
* fix optional field
* version
* limit number of in-flight subscriptions to 35000
* agent: optimize subscriptions memory usage more (do not store subscribed queues in memory) WIP
* use new session subscriptions data
* version
* remove old data structure
* remove version
* batch deletions
* test TSessionSubs
* comment
* smp server: store messages in PostgreSQL
* stored procedures to write and to expire messages
* function to export messages
* move all message functions to PostgreSQL, remove delete trigger
* comments
* import messages to db
* fix message import, add export
* fix export
* fix export
* fix compilation flags
* import messages line by line
* fix server start with database storage
* fix compilation
* comments
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)
* client certificates types (WIP)
* parameterize Transport
* protocol/schema/api changes
* agent API
* rename command
* agent subscriptions return local ClientServiceId to chat
* verify transmissions
* fix receiving client certificates, refactor
* ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions
* wait for subscriber with timeout
* safer
* refactor
* log
* remove unused
* WIP service subscriptions and associations, refactor
* process service subscriptions
* rename
* simplify switching subscriptions
* SMP service handshake with additional server handshake response
* notification delivery and STM persistence for services
* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error
* stats
* more stats
* rename SMP commands
* service subscriptions in ntf server agent (tests fail)
* fix
* refactor
* exports
* subscribe ntf server as service for associated queues
* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage
* INI option to support services
* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues
* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)
* fix test
* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription
* update RFC
* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands
* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission
* service errors, todos
* fix checkCredentials in ntf server, service errors
* refactor service auth
* refactor
* service agent: store returned queue count instead of expected
* refactor serverThread
* refactor serviceSig
* rename
* refactor, rename, test repeat NSUB service association
* respond with error to SUBS
* smp server: export/import service records between database and store log
* comment
* comments
* ghc 8.10.7
* smp server: short links and owners for channels
* types
* support mutliple rcv keys
* fix down migration, test/create server schema dump
* reduce schema dump
* parameterize type for link data by connection type
* return full connection link data
* test version
* change short link encoding
* test: print pg_dump output
* server pages, link encoding
* fix connection request when queue data and sender ID are created for old servers
* test, change pattern
* ci: install postgresql tools in runner (#1507)
* ci: install postgresql tools in runner
* ci: docker shell abort on error
* fix pattern for ghc 8.10.7
* patch ConnReqUriData SMP encoding to preserve queue mode after decoding
* test for RKEY
* fix/test store log with RKEY
---------
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
* smp protocol: short links types and other changes from RFC
* add fields for queue link ID and data
* create queue and ntf credentials with NEW command
* all tests
* simplfiy types, update rfc
* update rfc
* include SenderId in NEW request in case queue data is sent
* store queue data and generate link ID if needed
* update rfc
* agent API and types
* SMP commands and persistence for short links
* SMP client functions for short links
* agent client functions for short links
* create rcv queue with short link (TODO secret_box)
* encryption and encoding for link data, postgres client migration
* test creating short link
* get link and data, tests
* comments
* type signature
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary
* comments
* enable server_postgres flag by default, add CPP option to test
* refactor
* change default for server_postgres to False
* diff
* smp server: queue store typeclass
* parameterize JournalMsgStore
* typeclass for queue store
* postgres WIP
* compiles, passes tests
* remove StoreType
* split migrations
* progress
* addQueueRec
* reduce type spaghetti
* remove addQueue from typeclass definition
* getQueue
* test postgres storage in SMP server
* fix schema
* comment
* import queues to postgresql
* import queues to postgresql
* log
* fix test
* counts
* ci: test smp server with postgres backend (#1463)
* ci: test smp server with postgres backend
* postgres service
* attempt
* attempt
* empty
* empty
* PGHOST attempt
* PGHOST + softlink attempt
* only softlink attempt
* working attempt (PGHOST)
* remove env var
* empty
* do not start server without DB schema, do not import when schema exists
* export database
* enable all tests, disable two tests
* option for migration confirmation
* comments
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* smp server: journal store queue state validation
* validate state on update, refactor
* handle message parsing errors
* typo
* fix test, throw exception when file is smaller than expected
* fix
* fix test in ghc 8.10.7
* smp server: remove STM function from MsgStore
* polymorphic MsgStore
* jourmal storage for messages (WIP)
* more journal, test setup
* writeMsg
* test
* tryDelMsg
* delMsgQueue
* remove MsgStoreClass instance of existential wrapper for Msg stores
* store config
* extract common logic out of store instances
* add store type to config
* open journals, cache last message, tests pass
* CLI commands
* refactor import/export messages
* cli commands to import/export journal message store
* export journal without draining, import/export tests
* journal command
* import/export progress
* better progress info
* only log queue state once when importing
* logs
* handle IO errors in journal store, return as STORE error
* recover from state file errors
* fix message files after crash
* fix messages folder
* agent: support socks proxy without isolate-by-auth, with and without credentials
* add unit tests
* make xftp use correct SOCKS credentials
* rename
* support ipv6 in brackets, test parsing
* constant
* textToHostMode
* space
* SMP proxy: low level client and server implementation
* SMP proxy: server implementation (#1098)
* wip
* PRXY command
* progress
* SMP Proxy: client-level implementation (#1101)
* buildable
* encode messages
* update pkey
* fix queue types
* wrap SEND in proxy lookup
* WIP proxy client
* WIP
* post-rebase fixes
* encode something with something
* cleanup
* update
* fix nonce/corrId in batchingTests
* WIP: dig into createSMPProxySession
* agent
* test progress
* pass the test
* parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate
* include server certificate into THandle
* load server chain and sign key
* fix key type
* fix for 8.10
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* cleanup
* add 2-server test
* remove subsumed test
* checkCredentials for BrokerMsg
* skip batching tests
* remove userId param
* remove agent changes
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* remove unused type
* icrease test timeout
* reduce transport block
* envelope sizes
* don't fork unless have proxied commands to process
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* control: add delete command
* logDeleteQueue only when found
* use default StrEncoding for CPDelete arg
* move stats update from main transaction
* use size
* stabilize AUTH timing tests
* more iterations
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* agent: catch IO errors correctly in MonadError
* correction
* correction
* utils
* agentFinally to catch IO exceptions in ExceptT
* rename
* remove, inline
* rename utils
* utils unit test
* test to show catch and finally problems
* tryAllErrors
* enable all tests
* send "quota exceeded" message from SMP server when sender gets ERR QUOTA (ignored in the agent for now)
* send msg quota to the recipient to indicate that sender got ERR QUOTA, test
* switch between slow/fast retry intervals (tests do not pass yet)
* send QCONT message, refactor RetryInterval, test
* refactor
* remove comment
* remove space
* unit test for withRetryLock2
* refactor
* rfc: queue rotation
* update rfc
* messages for queue rotation
* allow multiple subscribed queues per connection in Agent/Client.hs
* refactor
* fix module name
* allow multiple queues in duplex connection type
* update commands
* add indices
* addConnectionRcvQueue
* switch connection to another queue (WIP)
* update schema/protocol
* switching queue works, but sending messages after the switch fails
* messages are delivered after rotation
* use connection-scoped queue ID
* rename queue records fields
* refactor using SMPQueue class/instances
* simplify queries
* QKEY: check queue is not secured, refactor
* update rfc
* mark queue as primary in QUSE
* queue rotation errors
* fix async ack
* fix async ACK to send OK
* correction
Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
* use SWCH command
* rename
* take into account only active queue subscription when determining connection result if at least one queue is active
* remove comment
* only enable notifications for connections with enableNtfs = True
* async test (WIP)
* async queue rotation test
* simplify combining results
* test with 2 servers
* fix unused subscribeConnection
* switch to cabal build
* increase build timeout
* increase delay in async test
* skip queue rotation tests
* build matrix
* step name
* use ubuntu-18.04 in build matrix
* enable rotation tests
Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
* test notification token with agent and notifications server
* notification server test with APNS mock
* set environment variables in the test
* use base64url encoding in encrypted notification data
