Commit Graph

238 Commits

Author SHA1 Message Date
spaced4ndy
1b8110a332 xftp server: restore file status from log (#1461)
* xftp server: restore file blocking info from log

* fix parse

* rework

* update

* rename
2025-02-21 22:43:56 +00:00
Evgeny
e78ab60c97 build: fix postgres (#1444) 2025-01-28 22:02:41 +00:00
Evgeny
6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
Evgeny Poberezkin
9404a3ab63 xftp server: block stats 2025-01-14 10:46:27 +00:00
Evgeny
3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
spaced4ndy
e557dc86e7 Merge branch 'master' into postgres 2024-12-20 17:18:54 +04:00
spaced4ndy
cf66aadc20 postgres: store implementation, conditional compilation (#1421)
* postgres: implementation wip

* to from field

* agent store compiles

* methods

* create store

* tests wip

* migration tests pass

* tests compile

* fix tests

* tests wip

* bool int

* tests wip

* tests wip

* more boolint

* more fixes

* more fields pass

* more fixes

* binary

* instances, binary

* test passes

* remove todos, more tests pass

* fix conflict

* fix bool

* fix sequence breaking

* fix insertedRowId

* skip ratchet re-synchronization tests

* after test

* file tests

* after test

* rename

* remove comment

* format

* remove unused

* suppress notices

* fixes

* move

* fix

* instance

* instance2

* fix

* instances

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-20 15:54:58 +04:00
Evgeny
0a827307d6 build: remove some modules from the client build (#1418)
* build: remove some modules from the client build

* remove websockets from client_library
2024-12-14 15:22:06 +00:00
spaced4ndy
64149599de postgres: db interfaces wip (sqlite passes) (#1419) 2024-12-12 17:42:58 +04:00
spaced4ndy
bef1e38295 xftp: add debug info to "too many recipients" error (#1410) 2024-12-04 19:56:33 +00:00
Evgeny
97104988a3 server: support server roles and operators (#1343)
* server: support server roles and operators

* make server operator optional

* allRoles

* fix test

* different server host in tests

* remove ServerCfg fields used only in UI

* comments

* choose different server for invitation when connecting via address

* fix test in ghc8107

* simplify
2024-11-22 16:43:37 +00:00
Evgeny
d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
Evgeny
ffecf200d4 build: client_library flag to build only used modules, remove package yaml (#1394)
* move dependencies to library

* remove package.yaml

* remove auto-generated comment

* cleanup target definitions

* add executable options

* add test rtsopts

* revert common options

* don't hide unused packages warning

* Add client_only cabal flag (#1396)

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-11-02 15:34:33 +00:00
Evgeny Poberezkin
9078118a6d Merge branch 'master' into journal-store 2024-10-25 15:43:04 +01:00
Evgeny
870a60fc5d smp server: reduce memory usage when starting, all servers: prevent losing state on termination while compacting state (#1385)
* smp server: reduce memory usage when starting

* servers: prevent losing state when compacting is incomplete (terminated abnormally)

* fix test

* fix/refactor

* message

* remove .bak

* simpler
2024-10-25 15:39:59 +01:00
Evgeny
0414ea59f0 smp server: journal message store (#1370)
* smp server: remove STM function from MsgStore

* polymorphic MsgStore

* jourmal storage for messages (WIP)

* more journal, test setup

* writeMsg

* test

* tryDelMsg

* delMsgQueue

* remove MsgStoreClass instance of existential wrapper for Msg stores

* store config

* extract common logic out of store instances

* add store type to config

* open journals, cache last message, tests pass

* CLI commands

* refactor import/export messages

* cli commands to import/export journal message store

* export journal without draining, import/export tests

* journal command

* import/export progress

* better progress info

* only log queue state once when importing

* logs

* handle IO errors in journal store, return as STORE error

* recover from state file errors

* fix message files after crash

* fix messages folder
2024-10-21 11:50:30 +01:00
Evgeny
f24ee70138 clients: do not send SNI (#1363) 2024-10-07 22:40:30 +01:00
Evgeny
0705d05078 smp: transport block encryption (#1317) 2024-10-06 18:05:06 +01:00
Evgeny
97d45fb2a6 smp server: pass server information via CLI during server initialization (#1356)
* smp server: pass server information via CLI during server initialization

* more info

* enable client expiration by default, disable port 8000

* update
2024-10-06 12:34:44 +01:00
Evgeny
50e4abf836 ntf server: control port (#1354)
* ntf server: control port

* version .405

* control

* use own_server_domains from INI file

* fix subs by server in control port

* bigger queues

* ntf server: only print subscriptions per own server when they are > 0

* fix tests

* revert version change

* dont import listThreads in ghc 8.10.7
2024-10-06 12:33:14 +01:00
Evgeny
2ab0c2a7c6 smp server: fix notifiers lost when compacting store log (#1347) 2024-10-01 22:28:01 +01:00
Evgeny
2a120dfe57 smp server: Allow serving HTTPS and transport on the same port (v2) (#1327)
* smp-server: Allow serving HTTPS and transport on the same port

* update rfc

* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started

* diff

* combine HTTP and SMP on the shared port

* Update to SockAddr

* Fix params and web.https parser

* Switch fork urls

* WIP: add smpServerTestStatic test

* Update warp-tls repo

* shared connection tests

* cleanup

* Add protocol tests

* rename cert file, enable both ports and web by default

* terminate with message on missing credentials

* test cert file

* client option to use port 443 as default SMP port

* use SNI in non-SMP clients

* supported

* remove TODO

* advice

* fix test build

* Add RSA-4096 check for web creds, fix test

* Remove directory listing from static app

* message

* messages

* update log tests

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-09-28 23:15:17 +01:00
Evgeny
3c18c4b66a servers: refactor TLS credentials (#1326)
* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started
2024-09-28 22:21:08 +01:00
Evgeny
22260cd719 agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes (#1321)
* agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes

* fix test
2024-09-22 22:22:05 +01:00
Evgeny
fa772af6c6 agent: support socks proxy without isolate-by-auth, with and without credentials (#1320)
* agent: support socks proxy without isolate-by-auth, with and without credentials

* add unit tests

* make xftp use correct SOCKS credentials

* rename

* support ipv6 in brackets, test parsing

* constant

* textToHostMode

* space
2024-09-15 21:36:31 +01:00
Evgeny
a70bd02c67 xftp server: round down file creation time to 1 hour (#1310) 2024-09-10 08:14:05 +01:00
Evgeny
dab1980d79 xftp: report receive file error with redirected file ID, when redirect is present (#1304)
* xftp: report receive file error with redirected file ID, when redirect is present

* fix test
2024-09-09 08:08:16 +01:00
Evgeny
a9e8d02593 server: bind control port server only to 127.0.0.1 for better security (in case of firewall misconfuguration) (#1280) 2024-09-05 13:48:09 +01:00
Evgeny
137ebc1cad servers: reduce memory used for period stats (#1298) 2024-09-02 23:12:08 +01:00
Evgeny
ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny
9596a03139 servers: reduce STM transactions (#1287)
* servers: reduce STM transactions

* switch stats and pending ENDs to IORef

* more IORef, split pending ENDs to use in one thread
2024-08-29 13:18:12 +01:00
Evgeny Poberezkin
aa60bd6770 use strict Maps, fix stats for sent END batches 2024-08-26 19:35:57 +01:00
Evgeny
7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny
f4d1a33ed8 agent: retry loop that resumes subscriptions as soon as agent is moved to foreground, suspend retry loops while agent is suspended (#1249)
* agent: retry loop that resumes subscriptions as soon as agent is moved to foreground, suspend retry loops while agent is suspended

* reset retry enterval when moving to foreground

* account for network state too

* simplify

* typo

* simplify
2024-08-05 15:09:46 +01:00
Evgeny Poberezkin
47ae921b86 use strict Map (#1241) 2024-07-23 08:27:54 +01:00
Evgeny Poberezkin
8d56b0ba85 agent: allow starting only XFTP sending workers (#1232)
* agent: allow starting only XFTP sending workers

* use Bool param

* flip bool

* update
2024-07-17 13:01:17 +01:00
spaced4ndy
ae8e1c5e9a agent: servers stats improvements, fixes (#1208)
* agent: reset stats startedAt time in memory

* getAgentSubsSummary

* change sub counting

* ack statistics

* add import

* instance

* Revert "instance"

This reverts commit 1f63740d56.

* Revert "add import"

This reverts commit ef72df8014.

* modify sub counting

* modify conn creation counting

* use int64

* file size stats

* remove import

* ack err counting

* conn del stats

* format

* new data

* add data

* toKB

* restore connCompleted

* use Int for counts

* use rq from scope

* remove getAgentSubsSummary

* fix connCompleted

* fix

* revert disabling stats

* use srv from scope

* combine ack stats

* modify

* comment

* count subs

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-07-03 12:36:15 +01:00
Evgeny Poberezkin
6a54a58a0d agent: remove legacy statistics (#1211)
* agent: remove legacy statistics

* delays after disposeAgent

* delays

* enable all tests

* more delays
2024-06-30 12:50:42 +01:00
spaced4ndy
c788692687 agent: servers summary types, api (#1202)
* agent: servers summary types, api [wip]

* encoding

* export

* Revert "export"

This reverts commit cd9f315fe8.

* comment

* rename

* simplify types

* uncomment

* comment

* rework

* comment, exports

* save, restore stats wip

* remove

* rename

* save stats periodically

* sigint, sigterm experiments

* corrections

* remove some proxy stats

* increase stat

* proposed stats

* fields

* Revert "sigint, sigterm experiments"

This reverts commit f876fbd418.

* wip

* retries -> attempts

* errs

* fix

* other errs

* more stat tracking

* sub stats

* remove xftp successes stats

* xftp stats tracking

* revert

* revert

* refactor

* remove imports

* comment

* Revert "refactor"

This reverts commit 26c368d82a.

* Revert "revert"

This reverts commit 4c9e3753b5.

* Revert "revert"

This reverts commit 6f65644053.

* todos

* persistence

* rename, fix

* config

* comment

* add started at to summary

* delete stats on user deletion

* reset api

* move

* getAgentServersSummary collect state logic

* corrections

* corrections

* remove

* rework

* decrease contention

* update

* more stats

* count sentProxied

* count subs

* remove unused

* comment

* remove comment

* comment

* export

* refactor

* cleanup

* intervals

* refactor

* refactor2

* refactor3

* refactor4

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-25 09:42:59 +04:00
spaced4ndy
da620c388a xftp: start chunk upload after all chunks are prepared (#1191) 2024-06-06 15:50:15 +04:00
spaced4ndy
3c0cd7efcc agent: separate type for agent file errors (#1185) 2024-06-05 18:44:32 +04:00
Evgeny Poberezkin
3d605310ed agent: remove protocol encodings and agent TCP server (#1189)
* rfc: remove agent protocol encodings

* agent: remove protocol encodings and agent TCP server

* update

* remove unused code

* remove

* move tests

* add delay to tests

* stabilize test

* test

* more delays

* reduce delays

* enable all tests

* delays

* style
2024-06-05 14:34:40 +01:00
Evgeny Poberezkin
3dab330480 use throwE instead of throwError (#1187)
* use throwE instead of throwError

* test delay
2024-06-05 11:20:50 +01:00
Evgeny Poberezkin
d28b17e787 xftp server: send HTTP2 error as timeout error to the client so it is treated as temporary (#1186)
* xftp server: log file reception error

* report HTTP2 error as timeout error

* reduce timeout to 5 min

* process timeout error in protocol response

* log warning on timeout/HTTP2 error
2024-05-31 22:18:28 +01:00
Evgeny Poberezkin
d12ea92055 agent: report correct errors from xftp handshake so they are treated as temporary (#1184)
* agent: report correct errors from xftp handshake so they are treated as temporary

* disable slow servers test

* remove comments

* all tests

* remove duplicate functions
2024-05-31 09:47:47 +01:00
Evgeny Poberezkin
5c2c88315a SMP server information (#1072)
* SMP server information

* fix tests

* country codes

* smp-server: serve contact and link pages from static files (#1084)

* smp-server: serve contact and link pages from static files

* generate index

* use params from ini

* render using ServerInformation

* tweak templates

* update

* fix some html

* smp-server: fix layout (#1097)

* smp-server: fix layout

* port fixes to link page

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* update server information page

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>

* update server info

* web: improve server info page design (#1166)

* web: improve server info page design

* web: fix font errors & some tags

* web: improve contact & invitation page layout and header

* update

* remove unused files/css

* cleanup

* fix link page

* remove unused font links

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* show contact address as is

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>
2024-05-29 11:30:42 +01:00
Evgeny Poberezkin
2ff5f5a832 agent: add context to CMD error (#1167)
* agent: add context to CMD error

* tests, more warnings

* fix tests

* log TBQueue sizes

* log locks

* more logs

* log sendMessagesB

* fix test

* log length

* refactor

* remove logging

* revert lock scope change

* cleanup

* add string error to A_PROHIBITED

* remove

* remove test limitations

* language
2024-05-24 14:13:01 +01:00
Evgeny Poberezkin
875a56b75a Merge branch 'stable' 2024-05-23 16:48:11 +01:00
Alexander Bondarenko
984394d906 core: remove MonadUnliftIO ExceptT orphans (#1169) 2024-05-23 16:44:00 +01:00
spaced4ndy
2e5433676e xftp: check proxy before downloading from unknown server (#1102)
* xftp: check proxy before downloading from unknown server

* corrections

* remove import

* tests

* comment

* tests

* don't wrap into internal error

* fix tests
2024-05-20 14:14:04 +01:00