Commit Graph

112 Commits

Author SHA1 Message Date
Evgeny Poberezkin
7712070cb3 Merge branch 'master' into proxy 2024-04-18 23:33:06 +01:00
Evgeny Poberezkin
c00c223f3b remove (or make optional) client key from handshakes (#1104)
* remove (or make optional) client key from handshakes

* remove comment
2024-04-18 22:43:49 +01:00
Evgeny Poberezkin
a3b229f668 SMP proxy: low level client and server implementation (#1096)
* SMP proxy: low level client and server implementation

* SMP proxy: server implementation (#1098)

* wip

* PRXY command

* progress

* SMP Proxy: client-level implementation (#1101)

* buildable

* encode messages

* update pkey

* fix queue types

* wrap SEND in proxy lookup

* WIP proxy client

* WIP

* post-rebase fixes

* encode something with something

* cleanup

* update

* fix nonce/corrId in batchingTests

* WIP: dig into createSMPProxySession

* agent

* test progress

* pass the test

* parameterize transport handle with transport peer to include server certificate (#1100)

* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>

* cleanup

* add 2-server test

* remove subsumed test

* checkCredentials for BrokerMsg

* skip batching tests

* remove userId param

* remove agent changes

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* remove unused type

* icrease test timeout

* reduce transport block

* envelope sizes

* don't fork unless have proxied commands to process

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-18 22:35:17 +01:00
Evgeny Poberezkin
2f43b43225 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-17 17:46:22 +01:00
Evgeny Poberezkin
ad4b5b6b71 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-15 13:47:48 +01:00
Alexander Bondarenko
875ddd80d6 SMP proxy: protocol (#954)
* WIP: proxy-related types

* test plan

* buildable with stubs

* add auth test

* update protocol

* fix

* update rfc

* update protocol/types

* disable test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-12 18:04:29 +01:00
Evgeny Poberezkin
1219446996 dispose agent instances, fix tests, (#1089)
* dispose agent instances in tests

* fix quota test

* tests: fix tests with -threaded (#1088)

* fix some tests

* match RTS opts with apps

* less verbose rts stats

* enable sqlite extended error codes

* clean up

* unfocus

* remove extendedErrorCode

It's actually setExtendedResultCodes, which isn't yet available.

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* fix switch test

* fix

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-04-10 19:34:02 +01:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Alexander Bondarenko
5e0123313c log control port commands, authentication for control port (#1071)
* log control port commands

* auth

* add auth to xftp, config and commands

* log missing auth

* put smp save under auth

* corrections

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-26 18:04:28 +00:00
Evgeny Poberezkin
ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00
Evgeny Poberezkin
abf956d425 fix tests for new protocol versions (#998)
* fix basic auth tests

* fix batching and syntax tests

* fix test matrix
2024-02-16 20:21:15 +00:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Alexander Bondarenko
cb64dabf75 tests: fix sporadic failures from reading stats (#975) 2024-01-30 11:36:07 +00:00
Evgeny Poberezkin
7f7a77c4eb SMP protocol: optimize batching transactions, remove Builder (#961)
* remove Builder

* fewer chunks

* remove lazy bytestrings

* optimize

* pad
2024-01-14 20:42:47 +00:00
Alexander Bondarenko
7c27357eb2 server: stop inactive clients without subscriptions, additional monitoring via control port (#901)
* fix thread labelling points

* add subscription thread labels

* add explicit sections for sockets/sessions/disconnects

* adjust label threads wrt. next blocking op

* WIP: emit event on server finish

So the event log wouldn't be cut short giving
false impression of unterminated sessions.

* add handshake timeout

* hack server socket stats

* trace closeTLS exceptions

* fix build

* clean up disconnectTransport

* make disconnectTransport terminating

Thus, participating in racyAny_ around it.

* trach snd/rcv time separately, log clients as CSV

* add direct client counting/stats

* add csv dump for socket threads

* guard socket threads dump from older GHC

* cut socket thread listing into socket-threads command

* store client creation time for stats

* wrap tls setup in timeout and labels

* server: expire clients without subscriptions

* fix check for subscriptions

* remove excess tracing

* move hardcoded timeouts to configs

* cleanup

* fix tests

* wrap direct queue waits in timeout

* rewrite old TCP wrappers with their SocketState variants

* add testInactiveWithSubs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-19 15:50:45 +00:00
Alexander Bondarenko
6bffcc8503 compatibility with GHC 8.10.7 (#906)
* build with ghc-8.10

* add 8.10 to matrix

* cleanup

* stats

* remove getField

* agent

* server

* remote

* space

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-11-29 12:53:06 +00:00
Moritz Angermann
002f36dde0 5.4.0.0: use ghc 9.6.2 (#791)
* Add 9.6 compat

* compile with GHC9.6.2: dependencies, imports, code

* refactor typeclasses

* refactor record dot

* update cabal version

* update github actions

* update direct-sqlcipher

* 5.4.0.0

* update cabal.project

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-08-24 20:22:02 +01:00
Evgeny Poberezkin
b001b748db agent: rename functions, remove unused code, stabilize tests (#830)
* agent: rename functions, stabilize tests

* test pattern match
2023-08-24 19:52:58 +01:00
Evgeny Poberezkin
44abb90c63 agent: fix async join to avoid sending HELLO again after restart (#829)
* agent: fix async join to avoid sending HELLO again after restart

* update test

* add callstack to test

* improve test
2023-08-21 14:35:46 +01:00
Evgeny Poberezkin
4fae7dcaee server: control port (#804)
* server: control port

* do not remove messages when saving via control port

* remove unused record fields

* fix tests
2023-07-15 13:33:00 +01:00
Evgeny Poberezkin
1901e96ecc server: make main SMP server queues unbounded (#802) 2023-07-14 21:07:45 +01:00
Evgeny Poberezkin
94540a2c71 servers: add TCP timeouts to avoid memory leaks (#776)
* servers: add TCP timeouts to avoid memory leaks

* fix tests

* only use RecvTimeOut

* servers: simple timeout for TCP transport

* revert dependency change

* simplify

* simplify

* simplify 2
2023-06-30 16:22:01 +01:00
Evgeny Poberezkin
4a927d1ae2 ntf server: better batching and logging (#780)
* ntf server: better batching and logging

* reduce batch delay for ntf server

* comments

* 5.1.3, ntf 1.4.2

* more logging

* more logging

* split large batches, more logging

* remove some logs
2023-06-26 20:14:35 +01:00
spaced4ndy
9f0b9a83d6 xftp: use chunk digest as entity id for transport session (#725) 2023-04-14 13:01:14 +04:00
Evgeny Poberezkin
44f0dd39f3 ci: disable 2 tests on linux only, switch to ubuntu 20 and 22 (#705)
* ci: disable 2 tests on linux only, switch to ubuntu 20 and 22

* fix platform name

* keep ubuntu 22 binaries

* Revert "keep ubuntu 22 binaries"

This reverts commit a1bbb12870.

* skip 1 more test

* skip 1 more test

* log os

* log os

* unconditionally skip test

* skip 1 more test in CI

* fix tests
2023-04-04 12:19:38 +01:00
Evgeny Poberezkin
5d8febc353 avoid possible race conditions when cancelled clients/asyncs can be removed after the new ones are added (so that the new are removed as well) (#627) 2023-01-30 17:46:06 +00:00
Evgeny Poberezkin
f4ad3a983e support users in agent to isolate traffic of different users (#598)
* users table, isolate traffic sessions by users or by queues

* remove extra indices

* corrections

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-11 17:47:20 +04:00
Evgeny Poberezkin
874f9f5cd6 option to enable/disable TLS handshake error logs (disable by default) (#581)
* option to enable/disable TLS handshake error logs (disable by default)

* refactor
2022-12-27 10:13:51 +00:00
Evgeny Poberezkin
e281efdcb8 server: SMP basic auth (#561)
* server: SMP basic auth

* update ini default

* rfc, types

* tests

* update INI file

* typo

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* refactor

* update INI options, log new queue creation mode on start (on/off/requires auth)

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-11-12 22:29:36 +00:00
Evgeny Poberezkin
374526a11b server: specialize monad (#557) 2022-11-03 11:47:28 +00:00
Evgeny Poberezkin
68138c08d2 support multiple server hostnames per queue (#494)
* store SMP client version per queue

* use versioned format for queue addresses (not compatible with v1)

* multiple server hosts in queue URI/data, versioned encoding (cross-version tests fail)

* choose server host based on socksProxy setting

* test works

* multiple server addresses mostly work

* add onion hosts for provided servers

* update SMP hosts
2022-08-12 11:32:04 +01:00
Evgeny Poberezkin
d788c3ca95 access servers via SOCKS proxy (#482)
* access servers via SOCKS proxy

* config to pass tcp timeout and option SOCKS5 proxy to the agent
2022-07-23 13:57:10 +01:00
Evgeny Poberezkin
6bbe1dfc66 use batch commands to resubscribe agent on server re-connection (#473) 2022-07-18 07:26:41 +01:00
Evgeny Poberezkin
a6f401041a SMP protocol v4: batching multiple server commands/responses in a transport block (#470)
* batch server commands in one transport block

* subscribe to multiple queues using batched commands

* agent method to subscribe to multiple queues using batched commands

* refactor

* test for batched subscriptions

* delete part of connections in batched test

* add resubscribeConnections

* remove comment

* update SMP protocol doc
2022-07-17 10:10:38 +01:00
Evgeny Poberezkin
cde8a11693 update stats logging and make it opt-in (#472)
* update stats logging and make it opt-in

* hSetBuffering

* update var name
2022-07-15 13:21:02 +01:00
Evgeny Poberezkin
9c1b43791c SMP v3: encrypt message timestamp and flags together with the body between server and recipient (#457)
* SMP v3: encrypt message timestamp and flags together with the body between server and recipient

* v3 tests

* update protocol doc

* add test for max size message

* delay in notifications test

* simplify v3

* encrypt server message to the recipient when sent

* refactor

* exit on error restoring the messages

* refactor, increase test timeout

* style

* add prints to the test

* remove error from unsafeMaxLenBS

* update protocol

* lint, improve test, change func param
2022-07-05 21:08:05 +01:00
Evgeny Poberezkin
bc26dc1d68 save and restore server stats on restart (#460) 2022-07-04 10:45:35 +01:00
Evgeny Poberezkin
91802fe94f Merge branch 'master' into f/notifications 2022-06-12 16:49:56 +01:00
Evgeny Poberezkin
0a71822dd0 v2.3.0: save and restore messages when the server is restarted (#395)
* save and restore messages when server is restarted

* read file line by line

* add import

* optmize restoring messages by reading the whole file

* update version to 2.3.0

* update scripts

* add script

* update readme
2022-06-12 15:59:14 +01:00
Evgeny Poberezkin
c1348aa54f SMP connection handshake v2 (#390)
* SMP connection handshake v2

* hadshake v2 [mostly] works

* all tests pass, some race conditions remain

* fix build

* fix race conditions, send CON after all HELLOs in duplexHandshake mode

* add comments

* comment

* add comments
2022-06-09 13:47:07 +01:00
Evgeny Poberezkin
ace94d7c69 log server stats (#367)
* log server stats

* separate stats updates from the existing transactions
2022-04-30 13:47:21 +01:00
Evgeny Poberezkin
1064e9c315 disconnect inactive clients (#366)
* disconnect inactive clients

* update test description

* use one thread per client to disconnect inactive clients

* remove comments

* more comments
2022-04-30 10:56:01 +01:00
Evgeny Poberezkin
60d0f19121 Merge branch 'master' into notifications-server 2022-04-20 08:23:26 +01:00
Evgeny Poberezkin
68f52c47dd message TTL and periodic expiration; setting to prevent creation of the new queues (#355)
* message TTL and periodic expiration; setting to prevent creation of the new queues

* fix expiration interval
2022-04-20 08:14:26 +01:00
Evgeny Poberezkin
45ddecc4b8 APNS/HTTP2 push client for notifications server (#350)
* APN push client (WIP)

* APNS push client works

* TODO comments

* comment

* send notification and process the response

* config

* e2e encryption for notification parameter

* read apns key filename and ID from env

* connect APNS client to server, fix notification JSON encoding to use dash case

* do not connect to APNS on start to pass CI tests

* fix CI test

* remove redundant import
2022-04-14 11:33:17 +01:00
Evgeny Poberezkin
fb26916eea ntf-server CLI, re-use SMP server CLI as a library (#347)
* ntf-server CLI, re-use SMP server CLI as a library

* add executable name
2022-04-03 10:37:32 +01:00
Evgeny Poberezkin
f466fa76e5 notifications: protocol, server (#335)
* notifications: protocol

* update protocol to include subscription ID and DH public key

* update protocol, started server

* add notification server subscription DH key

* use the same command type in notifications protocol, protocol parsing, server frame

* remove empty files
2022-03-27 08:36:00 +01:00
Evgeny Poberezkin
a37b24a8c2 use TCP keep-alive instead of SMP protocol PING (#330)
* use TCP keep-alive instead of SMP protocol PING

* update header files

* use CInt

* use Int again

* use network-3.1.2.7

* use https in cabal.project

* confitional keep-alive parameters to work on mac

* pass keep-alive opts via client/agent options

* remove space
2022-03-21 16:13:34 +00:00