Commit Graph

932 Commits

Author SHA1 Message Date
Evgeny Poberezkin
ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00
Alexander Bondarenko
246a0d10c2 xftp: raise internal upload limit to 5gb (#1020)
* xftp: raise internal upload limit to 5gb

* extract hard limit from agent
2024-03-02 18:46:05 +00:00
Evgeny Poberezkin
ce78646c7f refactor creating connection record (#1021) 2024-03-02 18:27:51 +00:00
spaced4ndy
294d7ec8dd agent: delay connection deletion to finish delivery of pending messages (#1015)
* agent: delay connection deletion to finish delivery of pending messages (wip)

* fixes, test

* notify, test

* add tests

* comment

* add test

* timeout

* test timeout

* up

* more tests

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-29 18:08:58 +00:00
Evgeny Poberezkin
c9ec7ea274 Merge branch 'stable' 2024-02-27 18:07:30 +00:00
Evgeny Poberezkin
5011ac2e7c Merge branch 'v5522' into stable 2024-02-27 17:58:08 +00:00
Alexander Bondarenko
2e3c961dab agent: optimize execution performance of slow memory operations during (re)subscriptions (#1013)
* agent: remove redundant set building

* tests: add tRcvQueuesTests (#1016)

* double accounting

* unit tests for invariants

* batch add queues

* ghc8107 compatibility

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-02-27 00:00:29 +00:00
spaced4ndy
050a921fbb agent: batch db operations for deleting connections and xftp files (#1009)
* agent: batch db operations for deleting connections

* batch delete rcv files

* snd files

* refactor

* refactor2

* lines

* refactor

* fix prefix path

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-26 10:28:50 +00:00
Evgeny Poberezkin
e2ec737c68 stop using public keys from table (#1014)
* stop using public keys from table

* comment
2024-02-25 16:19:04 +00:00
Alexander Bondarenko
95c0914809 agent: speed up SMP clientDisconnected (#1011) 2024-02-24 14:59:14 +00:00
Alexander Bondarenko
385463192b agent: extend worker summary (#1007)
* agent: extend worker summary with active/idle split

* simplify

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-24 13:29:36 +00:00
Evgeny Poberezkin
d7394aa762 fix ntf server version range in the client 2024-02-22 19:33:14 +00:00
Evgeny Poberezkin
0d843ea4ce Merge branch 'stable' 2024-02-21 20:33:06 +00:00
Evgeny Poberezkin
a34da38966 Merge branch 'stable' into v5521 2024-02-21 19:28:30 +00:00
Evgeny Poberezkin
5c1519df7f optimize: use IntMap (#1005) 2024-02-21 11:52:24 +00:00
Evgeny Poberezkin
03c24c3ada agent: fix performance leak, smp server: optimize and fix potential memory leak (#1004)
* fix agent performance leak when re-connecting clients, optimize SMP server

* let the clients remove themselves from their clientvars

* fix another test

* do not call removeSubs when AgentClient is not active

* revert some changes

* revert more, refactor

* comment

* rename

* refactor

* refactor

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-02-21 11:45:34 +00:00
Evgeny Poberezkin
9b38f69e7e smp server: add stats for queues deleted before and after they were secured (#1002) 2024-02-18 16:25:32 +00:00
Evgeny Poberezkin
89140e0ff0 use package version for all servers (#1001) 2024-02-18 08:56:35 +00:00
Evgeny Poberezkin
caeeb2df9c fix protocol in NTF server error 2024-02-17 15:32:44 +00:00
Evgeny Poberezkin
51be2fea20 agent: test NTF server (#1000) 2024-02-17 14:55:40 +00:00
Evgeny Poberezkin
f6acc5c240 remove test chunk after XFTP server test (#999) 2024-02-16 23:24:57 +00:00
Alexander Bondarenko
194a7bb58e agent: remove redirect entries when deleting rcv files (#995)
* agent: remove redirect entries when deleting rcv files

* update schema dump

* use rcv_file primary key for redirect lookup

* allow multiple redirects
2024-02-16 19:49:50 +04:00
Evgeny Poberezkin
c179073260 remove support for old versions (#990)
* remove support for old versions (WIP)

* fix

* updates

* use version var
2024-02-16 13:28:50 +00:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Evgeny Poberezkin
9254d8dac5 v5.5.3 2024-02-16 11:33:56 +00:00
Alexander Bondarenko
9ab34bca7d cli: add cert command to xftp and ntf servers (#991)
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-16 10:56:54 +00:00
spaced4ndy
6f62d7ff05 agent: add numRecipients parameter to send description (#993) 2024-02-15 13:24:46 +04:00
Alexander Bondarenko
7275714b8e cli: configure server paths from env (#992) 2024-02-14 20:31:06 +00:00
spaced4ndy
004597c764 agent: add index to file redirect migration (#988) 2024-02-14 18:47:39 +04:00
Alexander Bondarenko
2f7a288280 xftp: add sending and receiving via URI-encoded redirects (#968)
* xftp: add URI encoding for FileDescription

* tweak URI

* allow smaller blocks

* draft xftpReceiveFileFollow' and xftpSendFilePublic'

* add sending with redirect

* allow 64k chunks

* add migrations with redirect fields

* add test case

* fix deadlock

* revert CLI code

* WIP: working send/receive via URI

* fix field ambiguity

* cleanup

* update agent db schema

* update minimal chunk size

* add rfc

* apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add createRcvFileRedirect

* extract Simplex.Messaging.ServiceScheme and reuse for files

* update db schema

* check size/digest on receive complete

* cleanup

* use SIZE/DIGEST errors for redirects too

* split digest/size errors from redirect checks

* fix redirect error encoding

* rename RedirectMeta to RedirectFileInfo

* use query encoding for file URI

* group maybe fields under RcvFileRedirect

* add extras field

* update rfc

* add extras encoding and no-redirect tests

* fix toStrict for old ghc

* extra client data in file descr URI

* remove decoded yaml file

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-13 14:08:49 +00:00
Alexander Bondarenko
57e7c8ef6b smp-server: add cert CLI command to rotate online certificate (#984)
* smp-server: add gen-online CLI command

* use CN and algo from old certificate

* add cert checks to test

* rename command

* fix test

* cert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-12 20:17:08 +00:00
spaced4ndy
8de23c15ad agent: return ntf server in getNtfToken (#986) 2024-02-12 16:51:37 +04:00
Evgeny Poberezkin
e64b6cba4b 5.5.2.0 2024-02-10 22:07:02 +00:00
Evgeny Poberezkin
189885c50d protocol: ignore tail bytes in handshake blocks to allow future extension (#983)
* protocol: ignore tail bytes in handshake blocks to allow future extension

* cleanup
2024-02-05 21:01:20 +00:00
Evgeny Poberezkin
2ae1c9f79d agent: only retry connecting SMP clients when there are pending subscriptions (#981)
* agent: only retry connecting SMP clients when there are pending subscriptions

* fix

* remove retry on creating clients

* simplify
2024-02-01 16:17:37 +00:00
Alexander Bondarenko
24b84106a6 core: add activity tests to workers details (#980)
* core: add activity tests to workers details

* remove names

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-01 11:02:31 +00:00
Alexander Bondarenko
15bc027f23 core: fix reconnection bugs (#979)
* core: fix reconnection bugs

* untangle newProtocolClient

* refactor

* report busy clientVar error

* log error

* comments

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-01 10:55:50 +00:00
Evgeny Poberezkin
fd4eeb36db agent: optimize expired messages query (#976)
* agent: optimize expired messages query

* schema

* fix query

* fix

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix

* refactor

* comment

* refactor2

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-01-22 14:04:57 +00:00
Evgeny Poberezkin
1e49f1c92d 5.5.1.0 2024-01-19 19:47:39 +00:00
Alexander Bondarenko
f7cdec2f08 fix: support multiple notification servers in configuration (#971)
* tests: add trpl-keeps-server check

* add smp server switch check

* add connection test and fix withNtfServer

* Update src/Simplex/Messaging/Agent/NtfSubSupervisor.hs

Co-authored-by: Evgeny Poberezkin <e.poberezkin@me.com>

* use ntfServer from token

* rename

---------

Co-authored-by: Evgeny Poberezkin <e.poberezkin@me.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-01-19 19:42:28 +00:00
Evgeny Poberezkin
8ff89c19dc servers: do not keep stats file open during delays (#974) 2024-01-19 17:47:47 +00:00
Evgeny Poberezkin
baf2c47065 agent: expire messages failed after quota exceeded after 7 days, and expire multiple messages at once (#973)
* agent: expire messages failed after quota exceeded after 7 days, and expire multiple messages at once

* fix, test

* refactor

* catch in loop
2024-01-19 17:21:25 +00:00
Evgeny Poberezkin
b547f34cc0 Revert "Revert "agent: do not create user record in new databases (#957)" (#963)" (#964)
* Revert "Revert "agent: do not create user record in new databases (#957)" (#963)"

This reverts commit aee9088417.

* tests

* remove logging
2024-01-17 11:01:04 +00:00
Evgeny Poberezkin
f6ed4640d4 5.5.0.6 2024-01-16 17:07:15 +00:00
Evgeny Poberezkin
a0b35cec4f agent: fix potential race when good client can be removed instead of bad for the same transport session (#967)
* agent: fix potential race when good client can be removed instead of bad for the same transport session

* tryAgentError

* case
2024-01-16 13:45:51 +00:00
Evgeny Poberezkin
00c4ff4a21 agent: fix race condition in subscription worker (results in stuck delivery), better error handling in batch operations (#962)
* debugging test: join connection when reply queue creation fails

* more logs

* more logs

* more logs, fixed bug

* cleanup

* catch all errors

* simplify

* comment

* remove client ID counter
2024-01-16 10:29:35 +00:00
Evgeny Poberezkin
aee9088417 Revert "agent: do not create user record in new databases (#957)" (#963)
This reverts commit 9b9a0bd0df.
2024-01-14 22:28:28 +00:00
Evgeny Poberezkin
7f7a77c4eb SMP protocol: optimize batching transactions, remove Builder (#961)
* remove Builder

* fewer chunks

* remove lazy bytestrings

* optimize

* pad
2024-01-14 20:42:47 +00:00