Commit Graph

75 Commits

Author SHA1 Message Date
Evgeny Poberezkin
b11bb9f52e Merge branch 'master' into short-links 2024-09-09 16:01:23 +01:00
spaced4ndy
946e16339e agent: process last notification from list (#1307) 2024-09-09 16:42:14 +04:00
spaced4ndy
092ed088ca ntf: support for multiple messages encoding (#1305) 2024-09-09 16:03:17 +04:00
Evgeny Poberezkin
36b49b66f8 Merge branch 'master' into short-links 2024-09-05 15:10:25 +01:00
Evgeny
53e829a21c agent: store query errors, reduce slow query threshold to 1ms (#1277)
* agent: collect query errors stats

* simplify

* test

* use microseconds

* parens

* revert change to track all queries, reduce threshold to 1 ms
2024-08-24 14:51:26 +01:00
Evgeny
571d148bdf agent: do not start delivery workers when there are no messages to deliver (#1263)
* agent: use weak ThreadId and forkIO in workers instead of async (reduce memory)

* agent: do not start and exit delivery workers when there are no messages to deliver (#1264)

* agent: exit delivery workers when no messages to deliver

* only start delivery workers when there are pending messages

* fix

* focus test

* enable all tests

* lift

* do not exit workers when there is no work
2024-08-18 21:30:06 +01:00
Evgeny Poberezkin
4599dafa16 smp client: api to send/receive data blobs directly and via proxy, tests (#1246)
* smp client: api to send/receive data blobs directly and via proxy, tests

* fix test
2024-07-27 07:50:25 +01:00
Evgeny Poberezkin
051bf38bc7 agent: remove delays in notification processing, add notification stats (#1235)
* agent: remove delays in notification processing, add notification stats

* do not replace token after failed check

* more stats

* refactor

* fix

* backwards compatible JSON parsing

* retry deleting subscription on temporary error

* remove attempts to get multiple notification messages

* fix JSON decoding to be backwards compatible
2024-07-22 15:42:34 +01:00
spaced4ndy
7565ddd91c agent: initialize ratchet on processing confirmation to support decryption of messages received before allowConnection; return SndQueueSecured from joinConnection, acceptContact (#1233) 2024-07-18 19:54:14 +04:00
Evgeny Poberezkin
784d36d691 socks mode encoding (#1231)
* socks mode encoding

* test delays

* test delay

* unused import
2024-07-15 20:28:24 +01:00
Evgeny Poberezkin
d4fa0af350 ntf: additional tests for token registration when server and device are restarted (#1230)
* ntf: additional tests for token registration when server and device are restarted

* test response timeouts
2024-07-14 17:57:34 +01:00
Evgeny Poberezkin
743676421d ntf server: simplify and optimize subscriptions in server agent (#1219)
* increase queue size

* simplify

* refactor to optimize memory usage and performance

* comment

* refactor

* test delays
2024-07-07 21:17:12 +01:00
Evgeny Poberezkin
ce732c0efb agent: enable fast handshake (revert #1215) (#1216)
* Revert "agent: disable fast handshake (#1215)"

This reverts commit aa1d8d6c8b.

* remove import

* test delays
2024-07-03 18:05:27 +01:00
Evgeny Poberezkin
aa1d8d6c8b agent: disable fast handshake (#1215) 2024-07-02 13:40:37 +01:00
Evgeny Poberezkin
9ee684b0f4 rfc: faster handshake protocol (#1203)
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
2024-06-30 08:36:24 +01:00
Evgeny Poberezkin
3d605310ed agent: remove protocol encodings and agent TCP server (#1189)
* rfc: remove agent protocol encodings

* agent: remove protocol encodings and agent TCP server

* update

* remove unused code

* remove

* move tests

* add delay to tests

* stabilize test

* test

* more delays

* reduce delays

* enable all tests

* delays

* style
2024-06-05 14:34:40 +01:00
Evgeny Poberezkin
2ff5f5a832 agent: add context to CMD error (#1167)
* agent: add context to CMD error

* tests, more warnings

* fix tests

* log TBQueue sizes

* log locks

* more logs

* log sendMessagesB

* fix test

* log length

* refactor

* remove logging

* revert lock scope change

* cleanup

* add string error to A_PROHIBITED

* remove

* remove test limitations

* language
2024-05-24 14:13:01 +01:00
Evgeny Poberezkin
8097df5540 agent: persist server connection error (#1165)
* agent: persist server connection

* comment, refactor

* fix tests, reduce interval for ntf server

* cleanup

* 0
2024-05-22 13:25:49 +01:00
Evgeny Poberezkin
c4c983348f Merge branch 'master' into proxy 2024-05-13 14:13:52 +01:00
Evgeny Poberezkin
91cc48aabe agent: do not mark subscriptions on expired sessions as active, mark delayed subscriptions as active on the same session, do not cancel sending expired commands (#1127)
* agent: do not mark subscriptions on expired sessions as active, do mark delayed subscriptions as active on the same session, SUBOK response in the next SMP protocol version

* client: prevent sub actions from zombie sessions (#1122)

* client: prevent sub actions from zombie sessions

* error handling

* add AERR to pass background errors to client

* switch to activeClientSession

* put closeClient under activeClientSession

* rename

* remove AERR, do not skip processing

* move check and state update to one transaction

* catch extra UPs

* fix

* check queue is still pending before making it active

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* do not forward agent error

* revert not expiring sending subs

* fixes

* track subscription responses better

* add pending connection

* Revert "revert not expiring sending subs"

This reverts commit 4310a69391.

* do not expire sending commands

* rename

* fix race

* function

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-12 17:47:08 +01:00
Evgeny Poberezkin
6d60de2429 proxy: agent implementation (#1106)
* proxy: agent implementation

* revert change

* update rfc

* test stuck subscription mock

* store proxy sessions inside SMP client var

* rename

* create and use proxy session

* tests

* return proxy in SENT event

* rename, more tests

* rename

* more tests

* remove comment

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-01 08:48:33 +01:00
Evgeny Poberezkin
1219446996 dispose agent instances, fix tests, (#1089)
* dispose agent instances in tests

* fix quota test

* tests: fix tests with -threaded (#1088)

* fix some tests

* match RTS opts with apps

* less verbose rts stats

* enable sqlite extended error codes

* clean up

* unfocus

* remove extendedErrorCode

It's actually setExtendedResultCodes, which isn't yet available.

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* fix switch test

* fix

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-04-10 19:34:02 +01:00
Evgeny Poberezkin
84b8c8417b revert base64 change (#1081) 2024-04-03 09:27:36 +01:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Alexander Bondarenko
ee90ea6a69 replace base64-bytestring with base64 (#1065)
* replace base64-bytestring with base64

* minify

* use bytestring-0.10 compatible fork

PR pending...

* bump base64 fork with text compat

* move compat details to modules

* switch repo

* add back module

* cleanup

* minify

* clean imports

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-27 23:35:09 +00:00
Evgeny Poberezkin
b32259d048 notify about critical error on agent crash (#1062)
* notify about critical error on agent crash

* waitUntilActive

* disposeAgent

* fix
2024-03-21 14:54:57 +00:00
Evgeny Poberezkin
07fa75ec49 pqdr: agent api to confirm PQ encryption support during connection handshake, fix incorrect PQ support (#1032)
* pqdr: agent api to confirm PQ encryption support during connection handshake

* fix CONF, tests

* fix REQ, tests

* remove unused
2024-03-07 08:35:40 +00:00
Evgeny Poberezkin
e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00
Evgeny Poberezkin
155831ae36 tests: test notification decryption 2024-02-18 09:30:21 +00:00
Evgeny Poberezkin
51be2fea20 agent: test NTF server (#1000) 2024-02-17 14:55:40 +00:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Alexander Bondarenko
f7cdec2f08 fix: support multiple notification servers in configuration (#971)
* tests: add trpl-keeps-server check

* add smp server switch check

* add connection test and fix withNtfServer

* Update src/Simplex/Messaging/Agent/NtfSubSupervisor.hs

Co-authored-by: Evgeny Poberezkin <e.poberezkin@me.com>

* use ntfServer from token

* rename

---------

Co-authored-by: Evgeny Poberezkin <e.poberezkin@me.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-01-19 19:42:28 +00:00
Alexander Bondarenko
40fa34c2d5 tests: test multiple Ntf servers (#966)
* tests: add ntf case with multiple Ntf servers

* simplify test

* fix for master

* add server switch test

* add server switch test

* add message test for ntf server switch
2024-01-17 19:47:47 +00:00
Evgeny Poberezkin
00c4ff4a21 agent: fix race condition in subscription worker (results in stuck delivery), better error handling in batch operations (#962)
* debugging test: join connection when reply queue creation fails

* more logs

* more logs

* more logs, fixed bug

* cleanup

* catch all errors

* simplify

* comment

* remove client ID counter
2024-01-16 10:29:35 +00:00
Evgeny Poberezkin
ca527b4d6c use Builder for batching commands (#953)
* use Builder for batching commands

* refactor

* refactor

* increase timeout in test

* do not materialize builder

* use concat

* debug test

* debug2

* delays

* more delay, less worker delay

* remove logs

* reduce delay

* tests

* sized builder

* builder

* comment

* remove unsafe

* style

* remove fromIntegral
2024-01-08 22:54:27 +00:00
Evgeny Poberezkin
cf8b9c12ff remove test dependencies (#861)
* remove test dependencies

* .hlint.yaml
2023-10-22 10:50:36 +01:00
Evgeny Poberezkin
887ccbcf6c Merge branch 'master' into master-ghc9 2023-09-10 18:00:33 +01:00
Evgeny Poberezkin
84ce037867 extend SMP protocol to allow creating new queues without subscriptions (#839)
* Trace auto-subs flag

* Replace Bools with SubscriptionMode

* Handle SMOnlyCreate

* Wire remaining todos

* Update tests and fix

* Bump protocol level

* Apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

* Scrub needs_sub from agent DB

* Scrub a few more needSubs from the agent api

* change API, fix test

* agent: do not subscribe to queue when creating reply queue

* fix encoding

* WIP: SMOnlyCreate test

* Add SM guard for confirmQueue

Allows the test case to pump the allowConnection
reply without getting PROHIBITED.

* Remove tracing

* add noMessages, remove unnecessary getConnectionMessage from test

* add sending messages to the test

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2023-09-10 17:07:19 +01:00
Moritz Angermann
002f36dde0 5.4.0.0: use ghc 9.6.2 (#791)
* Add 9.6 compat

* compile with GHC9.6.2: dependencies, imports, code

* refactor typeclasses

* refactor record dot

* update cabal version

* update github actions

* update direct-sqlcipher

* 5.4.0.0

* update cabal.project

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-08-24 20:22:02 +01:00
Evgeny Poberezkin
b001b748db agent: rename functions, remove unused code, stabilize tests (#830)
* agent: rename functions, stabilize tests

* test pattern match
2023-08-24 19:52:58 +01:00
Evgeny Poberezkin
58cb2855d2 agent: delivery receipts (#752)
* rfc: delivery receipts

* update doc

* update rfc

* implementation plan, types, schema

* migration, update types

* update types

* rename migration

* export MsgReceiptStatus, JSON encoding

* update rfc, schema

* correction

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* skeleton of the implementation

* more implementation (some tests fail)

* more code, 1 test fails

* fix encoding

* refactor

* refactor

* test, fix

* only send receipts in v3+, test

* flip condition

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* flip condition

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* agent version 4 required to send receipts

* fix test

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-07-13 22:33:48 +01:00
Evgeny Poberezkin
acea477aab batch subscriptions for ntf server (#778)
* return updated ConnectionStats from switchConnectionAsync (#777)

* batch ntf server subscriptions to SMP servers

* refactor

* fix filtering queues in reconnect

* test, remove repeated grouping

* enable all tests

* reduce subscriptions count

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-06-25 08:27:50 +01:00
Evgeny Poberezkin
b1489138ea xftp: fix file reception on 32 bit platforms (#708) 2023-04-06 20:40:52 +01:00
Evgeny Poberezkin
44f0dd39f3 ci: disable 2 tests on linux only, switch to ubuntu 20 and 22 (#705)
* ci: disable 2 tests on linux only, switch to ubuntu 20 and 22

* fix platform name

* keep ubuntu 22 binaries

* Revert "keep ubuntu 22 binaries"

This reverts commit a1bbb12870.

* skip 1 more test

* skip 1 more test

* log os

* log os

* unconditionally skip test

* skip 1 more test in CI

* fix tests
2023-04-04 12:19:38 +01:00
Evgeny Poberezkin
6a665a0833 support down migrations (#696)
* support down migrations

* test

* update

* add down migrations and tests

* get current migrations

* corrections

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* update tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-03-27 12:44:53 +01:00
Evgeny Poberezkin
ddc2da8fe4 xftp: agent API for xftp commands and events (#684)
* xftp: agent API for xftp commands and events

* fix tests

* fix tests 2

* xftp: update agent send api

* update API to make temp path optional

* revert tmp path changes (fixes send)

---------

Co-authored-by: spacedandy <8711996+spaced4ndy@users.noreply.github.com>
2023-03-13 09:04:23 +00:00
Stanislav Dmitrenko
bccef0ba47 files: server and client spike - basic upload/download (#591)
* Files: main, env, stats, storeLog

* Better + transport

* Executable

* Env

* Update Client.hs, Server.hs, and 4 more files...

* Answer on request

* Delay

* Temp file

* Bypass cert check

* update package.yml, rename

* update store log

* extend HTTP2 transport

* refactor caStore

* HTTP2 body

* update server stats

* file server/client framework

* verify server commands

* process FNEW command, CLI test works

* simple XFTP server test (fails)

* fix test, refactor

* upload chunk works

* receive file chunk in the client

* remove transport handshake

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix names

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-02-13 13:36:02 +00:00
Evgeny Poberezkin
abee1fa70b Merge branch 'master' into users 2023-01-14 18:43:09 +00:00
Evgeny Poberezkin
701d06ba01 refactor tests to improve errors (#606)
* refactor tests to improve errors

* fix test descriptions

* fix test
2023-01-14 18:28:37 +00:00
Evgeny Poberezkin
f4ad3a983e support users in agent to isolate traffic of different users (#598)
* users table, isolate traffic sessions by users or by queues

* remove extra indices

* corrections

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-11 17:47:20 +04:00