Commit Graph

663 Commits

Author SHA1 Message Date
Evgeny
b3c8358a43 agent: combine connection deletion events (#1442) 2025-01-25 08:52:40 +00:00
Evgeny
eda9e36c82 agent: track queries (#1439) 2025-01-24 10:31:50 +00:00
Evgeny
6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
spaced4ndy
488c7082f3 agent: store interface (#1436) 2025-01-20 17:02:39 +04:00
spaced4ndy
fdde9863cd agent: add reopenStore function for postgres; run notification tests with postgres (#1435) 2025-01-17 16:27:37 +04:00
Evgeny
3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
spaced4ndy
9d9ec8cd0b agent: remove unused functions (#1432)
* agent: remove unused functions

* cleanup imports
2025-01-10 07:59:17 +00:00
Evgeny
992b42e922 agent: option to enable/disable vacuum after SQLite migration (#1429) 2024-12-28 21:06:42 +00:00
spaced4ndy
e557dc86e7 Merge branch 'master' into postgres 2024-12-20 17:18:54 +04:00
spaced4ndy
cf66aadc20 postgres: store implementation, conditional compilation (#1421)
* postgres: implementation wip

* to from field

* agent store compiles

* methods

* create store

* tests wip

* migration tests pass

* tests compile

* fix tests

* tests wip

* bool int

* tests wip

* tests wip

* more boolint

* more fixes

* more fields pass

* more fixes

* binary

* instances, binary

* test passes

* remove todos, more tests pass

* fix conflict

* fix bool

* fix sequence breaking

* fix insertedRowId

* skip ratchet re-synchronization tests

* after test

* file tests

* after test

* rename

* remove comment

* format

* remove unused

* suppress notices

* fixes

* move

* fix

* instance

* instance2

* fix

* instances

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-20 15:54:58 +04:00
Evgeny
4d640c16aa smp server: log prometheus metrics (#1411)
* smp server: log prometheus metrics

* save metrics

* diff

* lines

* version

* do not include Prometheus into client

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* add timestamp to metrics

* remove type

* remove version

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2024-12-18 11:19:13 +00:00
spaced4ndy
64149599de postgres: db interfaces wip (sqlite passes) (#1419) 2024-12-12 17:42:58 +04:00
Evgeny
97104988a3 server: support server roles and operators (#1343)
* server: support server roles and operators

* make server operator optional

* allRoles

* fix test

* different server host in tests

* remove ServerCfg fields used only in UI

* comments

* choose different server for invitation when connecting via address

* fix test in ghc8107

* simplify
2024-11-22 16:43:37 +00:00
Evgeny
21fbbf9106 smp server: update message counts during message expiration, increase idle interval (#1404)
* smp server: update message counts during message expiration, increase idle interval

* version

* fix

* flip results

* version
2024-11-17 09:44:40 +00:00
Evgeny
17a0be10fa smp server: expire messages in idle message queues (including not opened) (#1403)
* smp server: expire messages in idle message queues (including not opened)

* use message expiration interval

* simpler

* version

* remove version
2024-11-15 10:26:24 +00:00
Evgeny
d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
Evgeny
7c21945569 smp server: option to skip expiring messages on start, read queue state file end only (#1400) 2024-11-03 21:08:07 +00:00
Evgeny
ffecf200d4 build: client_library flag to build only used modules, remove package yaml (#1394)
* move dependencies to library

* remove package.yaml

* remove auto-generated comment

* cleanup target definitions

* add executable options

* add test rtsopts

* revert common options

* don't hide unused packages warning

* Add client_only cabal flag (#1396)

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-11-02 15:34:33 +00:00
Evgeny
5940514f40 smp server: remove queue from map when closing, test (#1392)
* smp server: remove queue from map when closing, test

* remove print

* refactor
2024-10-26 16:14:55 +01:00
Evgeny
56ca7bf30e smp server: do not log progress when server starts (#1390)
* smp server: do not log progress when server starts

* test
2024-10-26 10:33:18 +01:00
Evgeny Poberezkin
9078118a6d Merge branch 'master' into journal-store 2024-10-25 15:43:04 +01:00
Evgeny
870a60fc5d smp server: reduce memory usage when starting, all servers: prevent losing state on termination while compacting state (#1385)
* smp server: reduce memory usage when starting

* servers: prevent losing state when compacting is incomplete (terminated abnormally)

* fix test

* fix/refactor

* message

* remove .bak

* simpler
2024-10-25 15:39:59 +01:00
spaced4ndy
a8471eed5b ntf: get messages for multiple last notifications (#1378)
* ntf: get messages for multiple last notifications (#1352)

* ntf: separate get ntf conns api (#1379)

* ntf: separate get ntf conns api

* nonempty

* update

* update

* remove single get api

* fix test

* refactor

* refactor

* ntf: batch get connections (#1387)

* ntf: batch get apis

* works

* fix

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-25 15:39:41 +04:00
Evgeny
a3351ad060 smp server: fix export of journal storage (#1384)
* smp: fix export of journal storage

* catch errors when exporting messages
2024-10-25 09:11:47 +01:00
Evgeny
c9c075fd49 smp server: merge quota messages and set queue to "over quota" state after restoring, server tests with journal and memory store (#1380)
* smp: run server tests with journal and memory store

* merge quota messages, set queue to "over quota" state after restoring

* fix test
2024-10-23 09:17:23 +01:00
Evgeny
1484523164 smp server, journal store: check file size on append, validate queue state (#1377)
* smp server: journal store queue state validation

* validate state on update, refactor

* handle message parsing errors

* typo

* fix test, throw exception when file is smaller than expected

* fix

* fix test in ghc 8.10.7
2024-10-22 12:09:51 +01:00
Evgeny
dc17402896 smp server: updated journal store API (#1374)
* core: updated journal store API

* parameterize JournalState by read/write type

* collect stats when importing/expiring

* compare stored stats and store for messages and notifications

* simplify
2024-10-21 16:50:07 +01:00
Evgeny
0414ea59f0 smp server: journal message store (#1370)
* smp server: remove STM function from MsgStore

* polymorphic MsgStore

* jourmal storage for messages (WIP)

* more journal, test setup

* writeMsg

* test

* tryDelMsg

* delMsgQueue

* remove MsgStoreClass instance of existential wrapper for Msg stores

* store config

* extract common logic out of store instances

* add store type to config

* open journals, cache last message, tests pass

* CLI commands

* refactor import/export messages

* cli commands to import/export journal message store

* export journal without draining, import/export tests

* journal command

* import/export progress

* better progress info

* only log queue state once when importing

* logs

* handle IO errors in journal store, return as STORE error

* recover from state file errors

* fix message files after crash

* fix messages folder
2024-10-21 11:50:30 +01:00
Evgeny
8870442830 smp server: reject SKEY with different key earlier (at verification step), add tests (#1366) 2024-10-11 12:13:52 +01:00
spaced4ndy
9ad26c97a1 agent: prepareConnectionToAccept api (#1365)
* agent: prepare conn (plan)

* implement

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-11 11:05:14 +01:00
spaced4ndy
e1201f0967 agent: test db indexes (#1364) 2024-10-09 15:16:08 +04:00
Evgeny
f24ee70138 clients: do not send SNI (#1363) 2024-10-07 22:40:30 +01:00
Evgeny
f871f20172 smp server: fix notification delivery (#1350)
* .401

* stats for undelivered notifications

* logs, stats

* control port show ntf client IDs

* check that Ntf client is still current and that queue is not full, drop notifications otherwise

* prevent losing notifications when client is not current or queue full

* add log when no notifications, remove some logs

* reduce STM transaction

* revert version change
2024-10-07 09:01:28 +01:00
Evgeny
80d3518d55 tests: store log (#1348)
* 6.1.0.4

* tests: store log

* random

* more tests
2024-10-06 21:21:08 +01:00
spaced4ndy
49935473ae ntf: agent to notfiy MSGNTF on successful ack (#1355)
* ntf: agent to notfiy MSGNTF on successful ack

* move msgntf

* release lock earlier

* fix tests

* do not release get lock

* fix

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-06 19:48:22 +01:00
Evgeny
0705d05078 smp: transport block encryption (#1317) 2024-10-06 18:05:06 +01:00
Evgeny
97d45fb2a6 smp server: pass server information via CLI during server initialization (#1356)
* smp server: pass server information via CLI during server initialization

* more info

* enable client expiration by default, disable port 8000

* update
2024-10-06 12:34:44 +01:00
Evgeny
50e4abf836 ntf server: control port (#1354)
* ntf server: control port

* version .405

* control

* use own_server_domains from INI file

* fix subs by server in control port

* bigger queues

* ntf server: only print subscriptions per own server when they are > 0

* fix tests

* revert version change

* dont import listThreads in ghc 8.10.7
2024-10-06 12:33:14 +01:00
Evgeny Poberezkin
61b2b9df1a Re-commit: smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
This reverts commit 0ba3e69872.
2024-10-02 12:23:46 +01:00
Evgeny Poberezkin
0ba3e69872 Revert "smp server: persist notifications to avoid losing them when ntf server is offline (#1336)"
This reverts commit 0739f7b702.
2024-10-02 09:49:43 +01:00
spaced4ndy
243548631a ntf: repeat failed token deletions (#1346) 2024-10-01 16:30:14 +04:00
Evgeny
0739f7b702 smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
* ntf server: types for storing notifications

* fix tests

* remove comments

* batch NMSGs test

* fix test

* thread to expire notifications

* persist notifications on restart

* optimize

* refactor

* remove reverse

* control port stats
2024-09-29 18:50:16 +01:00
Evgeny
2a120dfe57 smp server: Allow serving HTTPS and transport on the same port (v2) (#1327)
* smp-server: Allow serving HTTPS and transport on the same port

* update rfc

* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started

* diff

* combine HTTP and SMP on the shared port

* Update to SockAddr

* Fix params and web.https parser

* Switch fork urls

* WIP: add smpServerTestStatic test

* Update warp-tls repo

* shared connection tests

* cleanup

* Add protocol tests

* rename cert file, enable both ports and web by default

* terminate with message on missing credentials

* test cert file

* client option to use port 443 as default SMP port

* use SNI in non-SMP clients

* supported

* remove TODO

* advice

* fix test build

* Add RSA-4096 check for web creds, fix test

* Remove directory listing from static app

* message

* messages

* update log tests

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-09-28 23:15:17 +01:00
Evgeny
3c18c4b66a servers: refactor TLS credentials (#1326)
* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started
2024-09-28 22:21:08 +01:00
Evgeny
7dcac19a67 xrcp: session encryption with forward secrecy (#1328)
* xrcp: session encryption with forward secrecy

* update doc

* accept keys as parameters

* docs: counter

* spec corrections
2024-09-24 09:22:26 +01:00
Evgeny
22260cd719 agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes (#1321)
* agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes

* fix test
2024-09-22 22:22:05 +01:00
Evgeny Poberezkin
d5a65656f5 test: socks proxy parameter encoding 2024-09-16 13:15:30 +01:00
Evgeny
fa772af6c6 agent: support socks proxy without isolate-by-auth, with and without credentials (#1320)
* agent: support socks proxy without isolate-by-auth, with and without credentials

* add unit tests

* make xftp use correct SOCKS credentials

* rename

* support ipv6 in brackets, test parsing

* constant

* textToHostMode

* space
2024-09-15 21:36:31 +01:00
Evgeny
7c25b3b1e0 smp protocol: send DELD when subscribed queue is deleted (#1312)
* smp protocol: send DELD when subscribed queue is deleted

* fix, test

* refactor

* send DELD event only if the client supports it (version 10); send END otherwise

* fix test

* notify on notifier rotation

* increase test delays
2024-09-11 13:16:51 +01:00
spaced4ndy
946e16339e agent: process last notification from list (#1307) 2024-09-09 16:42:14 +04:00