Commit Graph

162 Commits

Author SHA1 Message Date
Alexander Bondarenko
be9a84905f servers: read stores by chunks (#1080) 2024-04-01 19:42:30 +01:00
Alexander Bondarenko
a698f9a0c4 xftp-server: run expiration checks on start (#1042)
* xftp-server: run expiration checks on start

* rename

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-01 11:29:19 +01:00
Evgeny Poberezkin
39bb804fab xftp: fix repeated replica creation if it was in uploaded status (#1079)
* test with failing files (in progress)

* print

* add replica uploading state

* Revert "add replica uploading state"

This reverts commit 7068213aa6.

* <=

* fix

* prints

* test no redundancy

* all tests no redundancy

* revert delay

* refactor

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-04-01 13:37:35 +04:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Evgeny Poberezkin
44410535fd do not pass key to control port of xftp server (#1074) 2024-03-28 18:16:36 +00:00
Alexander Bondarenko
bbc9eccf4d xftp: prevent overwriting completed upload (#1063)
* xftp: prevent overwriting completed upload

* add size check for skipCommitted

* fix import

* fail on incorrect size

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-28 18:12:48 +00:00
Alexander Bondarenko
ee90ea6a69 replace base64-bytestring with base64 (#1065)
* replace base64-bytestring with base64

* minify

* use bytestring-0.10 compatible fork

PR pending...

* bump base64 fork with text compat

* move compat details to modules

* switch repo

* add back module

* cleanup

* minify

* clean imports

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-27 23:35:09 +00:00
Alexander Bondarenko
5e0123313c log control port commands, authentication for control port (#1071)
* log control port commands

* auth

* add auth to xftp, config and commands

* log missing auth

* put smp save under auth

* corrections

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-26 18:04:28 +00:00
Evgeny Poberezkin
2712fbc711 add key to XFTP server control port command (#1070)
* add key to XFTP server control port command

* name
2024-03-26 09:38:28 +00:00
Alexander Bondarenko
9eacae5492 xftp: remove duplicate check in receiveServerFile (#1057) 2024-03-20 15:50:38 +00:00
Alexander Bondarenko
d68ff7e22f xftp-server: set initial usedStorage from actual file records (#1045) 2024-03-19 13:53:02 +00:00
Evgeny Poberezkin
46e49fa823 xftp: return OK on duplicate upload attempt 2024-03-19 13:52:07 +00:00
Alexander Bondarenko
db3bddecca xftp-server: fix receiveServerFile (#1048)
* xftp-server: fix receiveServerFile

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-19 13:34:21 +00:00
Alexander Bondarenko
ca68eca86e agent: fix leak in getChunkDigest (#1051) 2024-03-15 12:18:15 +04:00
Alexander Bondarenko
ace09cc07d xftp: force single chunk for redirect descriptions (#1050)
* xftp: force single chunk for redirect descriptions

* Update src/Simplex/FileTransfer/Client/Main.hs

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-15 12:08:52 +04:00
spaced4ndy
e93ea6df71 xftp: fix sending large files (#1043) 2024-03-13 13:33:43 +04:00
Evgeny Poberezkin
ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00
Alexander Bondarenko
246a0d10c2 xftp: raise internal upload limit to 5gb (#1020)
* xftp: raise internal upload limit to 5gb

* extract hard limit from agent
2024-03-02 18:46:05 +00:00
spaced4ndy
050a921fbb agent: batch db operations for deleting connections and xftp files (#1009)
* agent: batch db operations for deleting connections

* batch delete rcv files

* snd files

* refactor

* refactor2

* lines

* refactor

* fix prefix path

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-26 10:28:50 +00:00
Evgeny Poberezkin
89140e0ff0 use package version for all servers (#1001) 2024-02-18 08:56:35 +00:00
Alexander Bondarenko
194a7bb58e agent: remove redirect entries when deleting rcv files (#995)
* agent: remove redirect entries when deleting rcv files

* update schema dump

* use rcv_file primary key for redirect lookup

* allow multiple redirects
2024-02-16 19:49:50 +04:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Evgeny Poberezkin
9254d8dac5 v5.5.3 2024-02-16 11:33:56 +00:00
Alexander Bondarenko
9ab34bca7d cli: add cert command to xftp and ntf servers (#991)
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-16 10:56:54 +00:00
spaced4ndy
6f62d7ff05 agent: add numRecipients parameter to send description (#993) 2024-02-15 13:24:46 +04:00
Alexander Bondarenko
2f7a288280 xftp: add sending and receiving via URI-encoded redirects (#968)
* xftp: add URI encoding for FileDescription

* tweak URI

* allow smaller blocks

* draft xftpReceiveFileFollow' and xftpSendFilePublic'

* add sending with redirect

* allow 64k chunks

* add migrations with redirect fields

* add test case

* fix deadlock

* revert CLI code

* WIP: working send/receive via URI

* fix field ambiguity

* cleanup

* update agent db schema

* update minimal chunk size

* add rfc

* apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add createRcvFileRedirect

* extract Simplex.Messaging.ServiceScheme and reuse for files

* update db schema

* check size/digest on receive complete

* cleanup

* use SIZE/DIGEST errors for redirects too

* split digest/size errors from redirect checks

* fix redirect error encoding

* rename RedirectMeta to RedirectFileInfo

* use query encoding for file URI

* group maybe fields under RcvFileRedirect

* add extras field

* update rfc

* add extras encoding and no-redirect tests

* fix toStrict for old ghc

* extra client data in file descr URI

* remove decoded yaml file

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-02-13 14:08:49 +00:00
Evgeny Poberezkin
e64b6cba4b 5.5.2.0 2024-02-10 22:07:02 +00:00
Evgeny Poberezkin
1e49f1c92d 5.5.1.0 2024-01-19 19:47:39 +00:00
Evgeny Poberezkin
f6ed4640d4 5.5.0.6 2024-01-16 17:07:15 +00:00
Evgeny Poberezkin
a0b35cec4f agent: fix potential race when good client can be removed instead of bad for the same transport session (#967)
* agent: fix potential race when good client can be removed instead of bad for the same transport session

* tryAgentError

* case
2024-01-16 13:45:51 +00:00
Evgeny Poberezkin
7f7a77c4eb SMP protocol: optimize batching transactions, remove Builder (#961)
* remove Builder

* fewer chunks

* remove lazy bytestrings

* optimize

* pad
2024-01-14 20:42:47 +00:00
Alexander Bondarenko
68f5e189a6 smp: check for progress when resubscribing (#956)
* smp: check for progress when resubscribing

* add allowClose to reconnectServer to distinguish entry points

* resolve error todo

* make reconnect async per-session

* remove allowClose

* deregister reconnecter when it finishes

* signal/react more work explicitly

* fix restart condition

* wait for reconnecter to finish

* remove redundant reconnect locks

* rename getClientVar for expanded scope

* formatting

* remove withPending

* move pending check to tryReconnectSMPClient loop

* combine pending check and slot release transactions

* actually reserve the slot for async

* simplify

* refactor

* refactor

* use removeClientVar

* rename

* refactor

* test

* reduce MonadError scope

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-12 19:29:18 +00:00
Evgeny Poberezkin
ad8cd1d515 5.5.0.5 2024-01-09 19:44:15 +00:00
Evgeny Poberezkin
ca527b4d6c use Builder for batching commands (#953)
* use Builder for batching commands

* refactor

* refactor

* increase timeout in test

* do not materialize builder

* use concat

* debug test

* debug2

* delays

* more delay, less worker delay

* remove logs

* reduce delay

* tests

* sized builder

* builder

* comment

* remove unsafe

* style

* remove fromIntegral
2024-01-08 22:54:27 +00:00
Evgeny Poberezkin
55808b0c82 5.5.0.4 2024-01-06 11:40:15 +00:00
Alexander Bondarenko
34056b9d7b xftp-server: add control port (#951)
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-05 17:36:19 +00:00
Evgeny Poberezkin
2488cf1e3a agent: restart worker when it terminated (#948)
* agent: restart worker when it terminated

* comment

* remove writeTMVar

* remove writeTMVar 2

* remove writeTMVar 3
2024-01-05 14:22:26 +00:00
Alexander Bondarenko
b4495bb4f0 xftp-server: fix stats (#944)
* xftp-server: fix stats

* count file expirations

* rename

* simplify

* check files size

* store and fix fileSize check

* do

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-05 11:30:27 +00:00
Evgeny Poberezkin
37d30240fd 5.5.0.3 2024-01-03 20:36:54 +00:00
Evgeny Poberezkin
d0588bd0ac 5.5.0.2 2023-12-27 20:35:56 +00:00
Alexander Bondarenko
7eb7bd5e81 xftp-server: add inactiveClientExpiration (#936)
* xftp-server: add inactiveClientExpiration

* fix test config

* add test

* add xftpPing

* switch to PCEUnexpectedResponse

* remove watchdog when server quits

* rename, loop

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-27 20:31:19 +00:00
spaced4ndy
22e1932372 xftp, ntf: refactor, reuse workers abstractions (#935)
* xftp: refactor, reuse workers abstractions (wip, test fails)

* rename

* refactor (tests pass)

* refactor2

* linebreaks

* do

* reuse worker abstractions for notifications

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-27 13:02:08 +04:00
spaced4ndy
e43e4860b9 xftp: file workers cycle through pending files based on retries count (limit number of iterations per work item to prevent stuck delivery) (#930)
* xftp: test file reception - shouldn't get stuck if file is deleted on server

* comment

* expiration test

* approach

* wip

* sort by retries in other works, revert some diff

* revert diff

* modify tests

* refactor

* refactor

* remove prints

* apply to other workers

* remove import

* comment

* refactor

* revert queue size

* fix test

* rename

* comment, correct number of retries

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-26 22:20:25 +00:00
Evgeny Poberezkin
a57066a826 5.5.0.1 2023-12-23 13:42:22 +00:00
Evgeny Poberezkin
2489333c87 agent: use DB to queue async commands and messages (#929)
* agent: use DB to queue async commands and messages

* fix message delivery, async commands dont work

* fix async commands

* remove comment

* rename

* comment

* balance agent operation

* empty lines

* balance another agent operation

* diff

* worker

* qAddress

* fix incorrect queue ID in v1

* types for queues

* fix test

* update index
2023-12-23 12:59:53 +00:00
Evgeny Poberezkin
fa457d1c25 agent: avoid race condition between worker and supervisor when getting work (#928) 2023-12-22 11:12:36 +00:00
Evgeny Poberezkin
13a60d1d39 use ChaChaDRG as the source of randomness (#920)
* use ChaChaDRG as the source of randomness

* remove functions using entropy directly

* comment
2023-12-21 00:12:08 +00:00
Evgeny Poberezkin
7627ce6b69 5.5.0-beta.0: SMP server/agent 5.5, XFTP server 1.2, NTF server 1.7 2023-12-19 16:07:11 +00:00