Commit Graph

162 Commits

Author SHA1 Message Date
spaced4ndy
d8e60ecfdb xftp: recoverable send - spike (#707) 2023-04-07 11:19:50 +04:00
Evgeny Poberezkin
9f8db13553 xftp: agent API to set and test servers (#704)
* xftp: agent API to set and test servers

* ProtocolTestStep

* update agent API for XFTP servers

* ci: update ubuntu versions

* disable test hanging on ubuntu
2023-04-05 20:37:03 +01:00
Evgeny Poberezkin
40da7e76dd import stateTVar 2023-03-18 13:57:17 +00:00
Evgeny Poberezkin
e33d5b5184 use different constraint for non-error functions (#690) 2023-03-17 16:01:24 +04:00
spaced4ndy
2b1a02d7d4 xftp: reconnect XFTP client on replica retry (#689)
* xftp: re-create XFTP client on replica retry

* closeXFTPSessionClient

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-03-17 09:50:49 +00:00
spaced4ndy
9255d48b69 xftp: move delay to replica (#688) 2023-03-16 18:12:38 +04:00
Evgeny Poberezkin
ddc2da8fe4 xftp: agent API for xftp commands and events (#684)
* xftp: agent API for xftp commands and events

* fix tests

* fix tests 2

* xftp: update agent send api

* update API to make temp path optional

* revert tmp path changes (fixes send)

---------

Co-authored-by: spacedandy <8711996+spaced4ndy@users.noreply.github.com>
2023-03-13 09:04:23 +00:00
Evgeny Poberezkin
8fde8e1344 xftp: agent command entities (#676)
* xftp: agent command entities

* progress event

* parameterize agent command by entity

* refactor

* Eq instance for APartyCmdTag
2023-03-10 09:10:52 +00:00
spacedandy
cc38deb65d Merge branch 'master' into xftp 2023-03-03 19:16:12 +04:00
spaced4ndy
cf147397a4 xftp: agent receive file (#637)
* xftp: agent receive file draft

* receiveFile more stubs, changes to types, schema

* cabal file

* comments

* xftp_server_id

* schema changes, get client, local worker/action

* agent env, save file description, adjust schema

* client stubs

* download chunk wip, store, schema, types

* remove commented code

* read file description, schema, types

* check received, decrypt

* remove pure

* todo

* add XFTP to agent client

* add user id

* agent test

* tests

* rename supervisor into agent

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-03-03 18:50:16 +04:00
Evgeny Poberezkin
2f7c115207 retryInterval functions pass current interval to the action, to resume with the same interval on restart (improve battery consumption) (#670) 2023-03-03 14:12:36 +00:00
Evgeny Poberezkin
bbf9e28696 xftp: add server to client encryption layer with cryptobox (#646)
* xftp: add server to client encryption layer with cryptobox

* workarounds for http2

* update http2 commit

* remove TODOs
2023-02-22 09:03:18 +00:00
Evgeny Poberezkin
15637e545b Merge branch 'master' into xftp 2023-02-17 21:40:35 +00:00
Evgeny Poberezkin
2ddfb044fc parameterize protocol by error type (#644) 2023-02-17 20:46:01 +00:00
spaced4ndy
4e8657097c xftp: cli client draft (#635)
* xftp: cli client draft

* more stubs

* compiles

* hash, app

* options parsers, random

* tmp

* xftp CLI client agent, simplify CLI command syntax

* only allow argument as a second parameter

* pivot signature draft

* receive file

* pivot sent chunks to recipients

* encryptFile - temp, chunks, specs

* send (upload) file and save file descriptions

* refactor, remove encrypted file

* save file size in description as string

* include filename inside padded encrypted file

* call chunk uploads concurrently, using queueing in HTTP2 as library client does not support concurrent streaming uploads

* download file (does not work yet)

* add digests to sent chunks

* fix recv - save file using AppendMode

* encrypt/decrypt sent file with secretbox

* remove print

* fix file description parsing in tests

* fix test

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-02-16 14:06:03 +04:00
Evgeny Poberezkin
eb1723acce refactor transport encoding, make client IO error "temporary" (#634)
* file server/client framework

* refactor

* make IOError temporary

* fix test
2023-02-10 19:06:28 +00:00
Evgeny Poberezkin
5d8febc353 avoid possible race conditions when cancelled clients/asyncs can be removed after the new ones are added (so that the new are removed as well) (#627) 2023-01-30 17:46:06 +00:00
Evgeny Poberezkin
b59669a65e refactor functions operating on single connections using batch functions (#620)
* refactor functions operating on single connections using batch functions

* update tests
2023-01-24 13:14:13 +00:00
Evgeny Poberezkin
d4fc638478 async batch connection deletion (#617)
* async batch connection deletion

* delete user record with connections, async connection deletion

* updates

* update query
2023-01-21 13:41:16 +00:00
Evgeny Poberezkin
a854b5f2b2 batched queue deletion (#611)
* batched queue deletion

* only re-connect client on subscribe errors, not in all batch commands

* add test, fix

* correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* refactor

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-19 16:53:28 +00:00
Evgeny Poberezkin
63b7956083 Merge branch 'master' into users 2023-01-18 22:20:29 +00:00
Evgeny Poberezkin
4f4b4f7eee only reconnect client on temporary errors in single subscription too (not currently used) (#616)
* only reconnect client on temporary errors in single subscription too (not currently used)

* refactor
2023-01-18 22:03:43 +00:00
Evgeny Poberezkin
f9c613ffb8 Merge branch 'master' into users 2023-01-18 21:18:42 +00:00
Evgeny Poberezkin
14cb88e725 remove completed async action handles from memory (#614)
* remove completed async action handles from memory

* name

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-18 18:22:17 +00:00
Evgeny Poberezkin
6ccbe5e66e retry unsuccessful subscriptions in case of temporary errors (#613)
* retry unsuccessful subscriptions in case of temporary errors

* do not send DOWN if connection has any active queues
2023-01-18 14:30:25 +00:00
Evgeny Poberezkin
324e01300d execute asynchronous commands for correct users, include session in messages from SMP clients to agent (#608)
* execute asynchronous commands for correct users

* pass transport session to message processing to avoid race condition

* account for server changes when determining unused servers

* enable one test

* enable all tests, remove log
2023-01-16 11:06:19 +00:00
Evgeny Poberezkin
b549910c17 support switching session mode, transport session now uses connection ID (#607)
* support switching session mode, transport session now uses connection ID

* refactor
2023-01-16 08:51:41 +00:00
Evgeny Poberezkin
9c9ba8c25c subscribe users in different sessions (#602)
* subscribe users in different sessions

* remove comments

* include userId to rcv queue map key

* use hash of userId[:entityId] as SOCKS proxy username
2023-01-13 09:58:39 +00:00
Evgeny Poberezkin
a0a092747a Merge branch 'master' into users 2023-01-12 16:37:59 +00:00
Evgeny Poberezkin
1f12697279 strict writes to STM, remove type class (#600) 2023-01-12 14:59:46 +00:00
Evgeny Poberezkin
f4ad3a983e support users in agent to isolate traffic of different users (#598)
* users table, isolate traffic sessions by users or by queues

* remove extra indices

* corrections

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-11 17:47:20 +04:00
Evgeny Poberezkin
acfa65200a deduplicate connections in connect/disconnect responses, log errors in tPut (#593)
* remove TODO for old handshake version (this HELLO is not sent now)

* deduplicate connections in responses and verify server in the list of subscribed queues

* log transport and LargeMsg in tPut (the results it returns are only used in the tests)

* refactor

* refactor
2023-01-06 17:14:49 +00:00
Evgeny Poberezkin
058e3ac55e send/process "quota exceeded" message from SMP server when sender gets ERR QUOTA (#585)
* send "quota exceeded" message from SMP server when sender gets ERR QUOTA (ignored in the agent for now)

* send msg quota to the recipient to indicate that sender got ERR QUOTA, test

* switch between slow/fast retry intervals (tests do not pass yet)

* send QCONT message, refactor RetryInterval, test

* refactor

* remove comment

* remove space

* unit test for withRetryLock2

* refactor
2023-01-04 14:10:13 +00:00
Evgeny Poberezkin
1d3a604988 refactor SMP client, NTF server store (#580)
* refactor SMP client, NTF server store

* remove undefined
2022-12-26 18:54:28 +00:00
Evgeny Poberezkin
267236a4f8 remove unnecessary exception handler, error format in stats 2022-12-26 13:00:41 +00:00
Evgeny Poberezkin
aa17cc55c1 collect agent stats (#579)
* collect agent stats

* remove comment
2022-12-26 12:02:20 +00:00
Evgeny Poberezkin
e4842f4f47 agent: include server address string into BROKER errors (#575)
* agent: include server address string into BROKER errors

* eol

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-11-30 08:25:07 +00:00
Evgeny Poberezkin
f3b6ed4db0 agent API for SMP server test (#563)
* agent API for SMP server test

* add JSON encoding
2022-11-15 07:50:46 +00:00
Evgeny Poberezkin
e281efdcb8 server: SMP basic auth (#561)
* server: SMP basic auth

* update ini default

* rfc, types

* tests

* update INI file

* typo

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* refactor

* update INI options, log new queue creation mode on start (on/off/requires auth)

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-11-12 22:29:36 +00:00
Evgeny Poberezkin
8d9816809f simplify queue rotation protocol (#550)
* simplify queue rotation protocol

* use simplified rotation protocol, update tests

* simplify schema

* delete all connection queues

* refactor

* switch notifications to the new queue, tests

* remove TODO

* refactor

* rfc correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* remove duplicate set active

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-10-31 13:33:28 +04:00
Evgeny Poberezkin
eb5c1c78cb connection queue redundancy and rotation (#521)
* rfc: queue rotation

* update rfc

* messages for queue rotation

* allow multiple subscribed queues per connection in Agent/Client.hs

* refactor

* fix module name

* allow multiple queues in duplex connection type

* update commands

* add indices

* addConnectionRcvQueue

* switch connection to another queue (WIP)

* update schema/protocol

* switching queue works, but sending messages after the switch fails

* messages are delivered after rotation

* use connection-scoped queue ID

* rename queue records fields

* refactor using SMPQueue class/instances

* simplify queries

* QKEY: check queue is not secured, refactor

* update rfc

* mark queue as primary in QUSE

* queue rotation errors

* fix async ack

* fix async ACK to send OK

* correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* use SWCH command

* rename

* take into account only active queue subscription when determining connection result if at least one queue is active

* remove comment

* only enable notifications for connections with enableNtfs = True

* async test (WIP)

* async queue rotation test

* simplify combining results

* test with 2 servers

* fix unused subscribeConnection

* switch to cabal build

* increase build timeout

* increase delay in async test

* skip queue rotation tests

* build matrix

* step name

* use ubuntu-18.04 in build matrix

* enable rotation tests

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-10-29 18:57:01 +01:00
Evgeny Poberezkin
19aef52135 debug agent locks 2022-10-22 16:43:14 +01:00
Evgeny Poberezkin
0fa40fbfb0 fix: connection-level locks (#543)
* fix: connection-level locks

* add context to store error

* convert SERatchetNotFound to CONN NOT_FOUND
2022-10-04 14:36:51 +01:00
Evgeny Poberezkin
f4baa5d672 increase concurrency with per-connection locks (#542)
* increase concurrency with per-connection locks

* only send CONNECT if the client wasnt connected already
2022-10-04 09:58:19 +01:00
Evgeny Poberezkin
29332a5e9f lints (#539)
* lints

* revert

* add import

* remove StrictData
2022-10-01 11:51:52 +01:00
Evgeny Poberezkin
413aad5139 allow passing agent store instead of path and key (#535) 2022-09-23 18:45:00 +01:00
Evgeny Poberezkin
13a9eee0cf cancel threads asynchronously (#527) 2022-09-17 00:18:58 +01:00
Evgeny Poberezkin
959ae34f16 fix agent not stopping and race conditions (#526)
* fix agent not stopping and race conditions

* simplify
2022-09-16 13:42:13 +01:00
Evgeny Poberezkin
afecefc3ad Merge branch 'master' into sqlcipher 2022-09-14 18:22:46 +01:00
JRoberts
e328ae5d06 closeAgentClient async commands clean up (#523) 2022-09-12 18:15:32 +01:00