Commit Graph

288 Commits

Author SHA1 Message Date
Evgeny Poberezkin 16d79da73e config 2024-05-15 15:15:59 +01:00
Evgeny Poberezkin 8516b0dd5b proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions (#1144)
* proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions

* rename

* inline

* comment

* use correct server version when encoding forwarded commands
2024-05-11 17:11:28 +01:00
Evgeny Poberezkin 9f72b249b4 Merge branch 'master' into proxy 2024-05-11 10:25:35 +01:00
Evgeny Poberezkin 727fd8b8f5 server: more efficient response to batched subscriptions (#1141)
* server: more efficient response to batched subscriptions

* add sndMsgQ for interleaving messages with replies

* remove redundant liftIO

* refactor

* refactor2

* rename

* fix

* diff

* remove comment

* remove comment

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-10 22:19:11 +01:00
Evgeny Poberezkin b48215d341 proxy: additional configuration for SOCKS proxy usage in SMP proxy client (#1138)
* proxy: additional configuration for SOCKS proxy usage in SMP proxy client

* update
2024-05-09 15:36:02 +01:00
Evgeny Poberezkin 6d471b8be2 Merge branch 'master' into proxy 2024-05-08 00:14:06 +01:00
Alexander Bondarenko 7a07076277 transport: require ALPN for extended handshakes (#1134)
* transport: require ALPN for extended handshakes

* fix 8.10 build

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-08 00:06:09 +01:00
Evgeny Poberezkin 7712070cb3 Merge branch 'master' into proxy 2024-04-18 23:33:06 +01:00
Evgeny Poberezkin a3b229f668 SMP proxy: low level client and server implementation (#1096)
* SMP proxy: low level client and server implementation

* SMP proxy: server implementation (#1098)

* wip

* PRXY command

* progress

* SMP Proxy: client-level implementation (#1101)

* buildable

* encode messages

* update pkey

* fix queue types

* wrap SEND in proxy lookup

* WIP proxy client

* WIP

* post-rebase fixes

* encode something with something

* cleanup

* update

* fix nonce/corrId in batchingTests

* WIP: dig into createSMPProxySession

* agent

* test progress

* pass the test

* parameterize transport handle with transport peer to include server certificate (#1100)

* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>

* cleanup

* add 2-server test

* remove subsumed test

* checkCredentials for BrokerMsg

* skip batching tests

* remove userId param

* remove agent changes

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* remove unused type

* icrease test timeout

* reduce transport block

* envelope sizes

* don't fork unless have proxied commands to process

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-18 22:35:17 +01:00
Alexander Bondarenko 875ddd80d6 SMP proxy: protocol (#954)
* WIP: proxy-related types

* test plan

* buildable with stubs

* add auth test

* update protocol

* fix

* update rfc

* update protocol/types

* disable test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-12 18:04:29 +01:00
Alexander Bondarenko be9a84905f servers: read stores by chunks (#1080) 2024-04-01 19:42:30 +01:00
Evgeny Poberezkin 6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Alexander Bondarenko 5e0123313c log control port commands, authentication for control port (#1071)
* log control port commands

* auth

* add auth to xftp, config and commands

* log missing auth

* put smp save under auth

* corrections

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-26 18:04:28 +00:00
Evgeny Poberezkin ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00
Evgeny Poberezkin dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00
Evgeny Poberezkin 0d843ea4ce Merge branch 'stable' 2024-02-21 20:33:06 +00:00
Evgeny Poberezkin a34da38966 Merge branch 'stable' into v5521 2024-02-21 19:28:30 +00:00
Evgeny Poberezkin 5c1519df7f optimize: use IntMap (#1005) 2024-02-21 11:52:24 +00:00
Evgeny Poberezkin 03c24c3ada agent: fix performance leak, smp server: optimize and fix potential memory leak (#1004)
* fix agent performance leak when re-connecting clients, optimize SMP server

* let the clients remove themselves from their clientvars

* fix another test

* do not call removeSubs when AgentClient is not active

* revert some changes

* revert more, refactor

* comment

* rename

* refactor

* refactor

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-02-21 11:45:34 +00:00
Evgeny Poberezkin 9b38f69e7e smp server: add stats for queues deleted before and after they were secured (#1002) 2024-02-18 16:25:32 +00:00
Evgeny Poberezkin c179073260 remove support for old versions (#990)
* remove support for old versions (WIP)

* fix

* updates

* use version var
2024-02-16 13:28:50 +00:00
Evgeny Poberezkin 416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Alexander Bondarenko 9ab34bca7d cli: add cert command to xftp and ntf servers (#991)
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-16 10:56:54 +00:00
Alexander Bondarenko 7275714b8e cli: configure server paths from env (#992) 2024-02-14 20:31:06 +00:00
Alexander Bondarenko 57e7c8ef6b smp-server: add cert CLI command to rotate online certificate (#984)
* smp-server: add gen-online CLI command

* use CN and algo from old certificate

* add cert checks to test

* rename command

* fix test

* cert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-12 20:17:08 +00:00
Alexander Bondarenko 8fd5e9f25a servers: make writeStoreLogRecord atomic (#946)
* servers: make writeStoreLogRecord atomic

* comment

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-03 20:29:09 +00:00
Alexander Bondarenko 36298f2cea smp-server: update stats (#943)
* smp-server: check queue balance in stats vs store

* smp-server: add msgExpired stats

* add msgExpired stats

* split expire/stats transactions

* count and pass msgExpired explicitly

* save/load qCount and use it for checking store
2024-01-02 21:24:17 +00:00
Alexander Bondarenko efe7ce27e7 control: add delete command (#933)
* control: add delete command

* logDeleteQueue only when found

* use default StrEncoding for CPDelete arg

* move stats update from main transaction

* use size

* stabilize AUTH timing tests

* more iterations

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-26 20:20:12 +00:00
Evgeny Poberezkin 13a60d1d39 use ChaChaDRG as the source of randomness (#920)
* use ChaChaDRG as the source of randomness

* remove functions using entropy directly

* comment
2023-12-21 00:12:08 +00:00
Evgeny Poberezkin 8c250ebe19 agent: batch sending messages (#922)
* agent: batch sending messages (attempt 4)

* handle errors in batch sending

* batch attempt 5 (#923)

* attempt 5

* remove IORefs

* add liftA2 for 8.10 compat

* remove db-related zipping

* traversable

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>

* s/mapE/bindRight/

* name

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* comment

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* remove unused funcs

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-12-19 23:01:34 +00:00
Alexander Bondarenko 7c27357eb2 server: stop inactive clients without subscriptions, additional monitoring via control port (#901)
* fix thread labelling points

* add subscription thread labels

* add explicit sections for sockets/sessions/disconnects

* adjust label threads wrt. next blocking op

* WIP: emit event on server finish

So the event log wouldn't be cut short giving
false impression of unterminated sessions.

* add handshake timeout

* hack server socket stats

* trace closeTLS exceptions

* fix build

* clean up disconnectTransport

* make disconnectTransport terminating

Thus, participating in racyAny_ around it.

* trach snd/rcv time separately, log clients as CSV

* add direct client counting/stats

* add csv dump for socket threads

* guard socket threads dump from older GHC

* cut socket thread listing into socket-threads command

* store client creation time for stats

* wrap tls setup in timeout and labels

* server: expire clients without subscriptions

* fix check for subscriptions

* remove excess tracing

* move hardcoded timeouts to configs

* cleanup

* fix tests

* wrap direct queue waits in timeout

* rewrite old TCP wrappers with their SocketState variants

* add testInactiveWithSubs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-19 15:50:45 +00:00
Evgeny Poberezkin 9d12d76078 add fourmolu (#868)
* add fourmolu

* linebreak

* simplify

* comment
2023-10-22 09:20:14 +01:00
Evgeny Poberezkin 7bdae793cb server: add control port commands for clients and ghc threads (#836)
* server: add control port commands for clients and ghc threads (#835)

* Add stats-rts control query

With supporting ghc-options that would provide the data.

* Add CPSkip command

Allows spamming empty lines a few times to clean up the view.

* server: Add CP commands to enumerate clients and threads

* style

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

* use base64 encoding for session ID

* fromMaybe

* whitespace

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2023-08-26 16:02:18 +01:00
Evgeny Poberezkin 4fae7dcaee server: control port (#804)
* server: control port

* do not remove messages when saving via control port

* remove unused record fields

* fix tests
2023-07-15 13:33:00 +01:00
Evgeny Poberezkin d989d11478 server: refactor restoring store log (#803) 2023-07-15 10:37:08 +01:00
Evgeny Poberezkin 1901e96ecc server: make main SMP server queues unbounded (#802) 2023-07-14 21:07:45 +01:00
Evgeny Poberezkin 94540a2c71 servers: add TCP timeouts to avoid memory leaks (#776)
* servers: add TCP timeouts to avoid memory leaks

* fix tests

* only use RecvTimeOut

* servers: simple timeout for TCP transport

* revert dependency change

* simplify

* simplify

* simplify 2
2023-06-30 16:22:01 +01:00
Evgeny Poberezkin 4a927d1ae2 ntf server: better batching and logging (#780)
* ntf server: better batching and logging

* reduce batch delay for ntf server

* comments

* 5.1.3, ntf 1.4.2

* more logging

* more logging

* split large batches, more logging

* remove some logs
2023-06-26 20:14:35 +01:00
Evgeny Poberezkin a5a3a2cbad server: increase server main queue size 2023-05-27 15:18:53 +01:00
Evgeny Poberezkin 9d83a9c017 configure message/file expiration time in INI file (#749)
* configure message/file expiration time in INI file

* correct comment

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-05-09 20:07:42 +01:00
spaced4ndy bdff274f56 use Int64 for time calculations (#706) 2023-04-04 14:51:13 +04:00
Evgeny Poberezkin 3ab5e9d110 xftp: expire files on the server, track/limit used storage (#651)
* xftp: expire files on the server

* track/limit used storage

* support storage quota and disabling queue creation in CLI parameters

* fix ini file

* correction

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-02-24 11:09:51 +04:00
Evgeny Poberezkin b342b1dc59 extend HTTP2 transport (#632)
* extend HTTP2 transport

* refactor caStore

* HTTP2 body

* enable test

* remove maxBodySize
2023-02-09 23:09:24 +00:00
Evgeny Poberezkin c4f377a85b fix SMP server stats (#612)
* fix SMP server stats

* add server stats test

* fix test
2023-01-18 11:07:25 +00:00
Evgeny Poberezkin 56cc2bc71f additional SMP server stats (#605)
* additional SMP server stats

* refactor
2023-01-14 13:23:37 +00:00
Evgeny Poberezkin 1f12697279 strict writes to STM, remove type class (#600) 2023-01-12 14:59:46 +00:00
Evgeny Poberezkin 0dddd59b36 increase server internal queue sizes (#595) 2023-01-11 16:14:33 +00:00
Evgeny Poberezkin 61e0c346df simplify C.sign, Crypto tests, triage TODOs (#592)
* simplify C.sign, Crypto tests, triage TODOs

* update condition

* remove todos

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-06 15:34:05 +00:00
Evgeny Poberezkin 058e3ac55e send/process "quota exceeded" message from SMP server when sender gets ERR QUOTA (#585)
* send "quota exceeded" message from SMP server when sender gets ERR QUOTA (ignored in the agent for now)

* send msg quota to the recipient to indicate that sender got ERR QUOTA, test

* switch between slow/fast retry intervals (tests do not pass yet)

* send QCONT message, refactor RetryInterval, test

* refactor

* remove comment

* remove space

* unit test for withRetryLock2

* refactor
2023-01-04 14:10:13 +00:00