Commit Graph

71 Commits

Author SHA1 Message Date
Evgeny
1329fc726f smp: support client notices (#1659)
* agent: support client notices

* improve

* fix, test

* rename

* cleanup

* send and process notices in more cases

* dont delete

* dont remove notice on other permanent errors

* dont remove notice if there is no notice ID in queue

* add server to error

* allow deleting

* only use notice if key hash matches
2025-10-17 18:34:59 +01:00
Evgeny
e48bedeaf2 servers: fix server pages when source code is not specified, include commit SHA in printed version and in web page. (#1608)
* smp server: fix server pages when source code is not specified

* servers: include git commit in version

* flexible alpn

* fix test

* fix ghc 8.10.7 build
2025-08-23 19:59:00 +01:00
Evgeny
f4e7469f96 xftp server: prometheus metrics (#1595)
* xftp server: prometheus metrics

* update

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-07-30 16:18:49 +01:00
Evgeny
c0f357d817 servers: control port session improvements (#1591)
* servers: prohibit changing role during control port session

* quota for blocked queues

* allow disabling blocking and quota

* fix test

* fix INI file
2025-07-21 15:19:43 +01:00
Evgeny
da37384335 smp server: batch commands (#1559)
* protocol: refactor types and encoding

* clean

* smp server: batch commands (#1560)

* smp server: batch commands verification into one DB transaction

* ghc 8.10.7

* flatten transmission tuples

* diff

* only use batch logic if there is more than one transmission

* func

* reset NTF service when adding notifier

* version

* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"

This reverts commit 3df2425162.

* version

* Revert "version"

This reverts commit d80a6b74c5.
2025-06-12 23:05:04 +01:00
Evgeny
5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny
8e86c97a13 servers: include supported ALPNs in server transport config (#1557) 2025-06-03 13:35:35 +01:00
Evgeny
53b72469b6 refactor types (#1551) 2025-05-27 22:31:02 +01:00
Evgeny
0a3d014f5d ntf server: priority db pool for notifications and client commands, concurrent resubscriptions, CLI options (#1529)
* ntf server: option to skip specific tokens when importing store log

* logs

* batch later

* subscribe in parallel, bigger queues

* ntf server: test schema and migrations

* optimize subscriptions

* log level

* resubscribe before starting

* better subscribing

* pooled concurrent resubscriptions

* 5

* async resubscribe

* typo

* priority pool

* db batch size
2025-04-30 11:40:32 +01:00
spaced4ndy
1b8110a332 xftp server: restore file status from log (#1461)
* xftp server: restore file blocking info from log

* fix parse

* rework

* update

* rename
2025-02-21 22:43:56 +00:00
Evgeny
6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
Evgeny
3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
Evgeny
d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
Evgeny
0705d05078 smp: transport block encryption (#1317) 2024-10-06 18:05:06 +01:00
Evgeny
50e4abf836 ntf server: control port (#1354)
* ntf server: control port

* version .405

* control

* use own_server_domains from INI file

* fix subs by server in control port

* bigger queues

* ntf server: only print subscriptions per own server when they are > 0

* fix tests

* revert version change

* dont import listThreads in ghc 8.10.7
2024-10-06 12:33:14 +01:00
Evgeny
3c18c4b66a servers: refactor TLS credentials (#1326)
* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started
2024-09-28 22:21:08 +01:00
Evgeny
a70bd02c67 xftp server: round down file creation time to 1 hour (#1310) 2024-09-10 08:14:05 +01:00
Evgeny
a9e8d02593 server: bind control port server only to 127.0.0.1 for better security (in case of firewall misconfuguration) (#1280) 2024-09-05 13:48:09 +01:00
Evgeny
ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny
9596a03139 servers: reduce STM transactions (#1287)
* servers: reduce STM transactions

* switch stats and pending ENDs to IORef

* more IORef, split pending ENDs to use in one thread
2024-08-29 13:18:12 +01:00
Evgeny Poberezkin
aa60bd6770 use strict Maps, fix stats for sent END batches 2024-08-26 19:35:57 +01:00
Evgeny
7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny Poberezkin
3dab330480 use throwE instead of throwError (#1187)
* use throwE instead of throwError

* test delay
2024-06-05 11:20:50 +01:00
Evgeny Poberezkin
d28b17e787 xftp server: send HTTP2 error as timeout error to the client so it is treated as temporary (#1186)
* xftp server: log file reception error

* report HTTP2 error as timeout error

* reduce timeout to 5 min

* process timeout error in protocol response

* log warning on timeout/HTTP2 error
2024-05-31 22:18:28 +01:00
Evgeny Poberezkin
d12ea92055 agent: report correct errors from xftp handshake so they are treated as temporary (#1184)
* agent: report correct errors from xftp handshake so they are treated as temporary

* disable slow servers test

* remove comments

* all tests

* remove duplicate functions
2024-05-31 09:47:47 +01:00
Evgeny Poberezkin
2ff5f5a832 agent: add context to CMD error (#1167)
* agent: add context to CMD error

* tests, more warnings

* fix tests

* log TBQueue sizes

* log locks

* more logs

* log sendMessagesB

* fix test

* log length

* refactor

* remove logging

* revert lock scope change

* cleanup

* add string error to A_PROHIBITED

* remove

* remove test limitations

* language
2024-05-24 14:13:01 +01:00
Evgeny Poberezkin
b27f126bab include server version range in transport handle (#1135)
* include server version range in transport handle

* xftp handshake

* remove coment

* simplify

* comments
2024-05-08 23:00:00 +01:00
Evgeny Poberezkin
c00c223f3b remove (or make optional) client key from handshakes (#1104)
* remove (or make optional) client key from handshakes

* remove comment
2024-04-18 22:43:49 +01:00
Evgeny Poberezkin
2f43b43225 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-17 17:46:22 +01:00
Evgeny Poberezkin
fbc25b171e xftp: reply ERR AUTH when file is removed from server storage (#1091) 2024-04-10 23:02:48 +01:00
Alexander Bondarenko
4c20ff6d00 xftp: negotiate protocol with ALPN (#1047)
* xftp: negotiate protocol with ALPN

* add RFC

* add handshake implementation

* implement extended handshake

* enable authentication

* update rfc

* Apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* cleanup

* discard trailing data

* cleanup diff

* use find

* rename

* refactor

* add x509 tests

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-09 15:03:40 +01:00
Evgeny Poberezkin
84b8c8417b revert base64 change (#1081) 2024-04-03 09:27:36 +01:00
Alexander Bondarenko
a698f9a0c4 xftp-server: run expiration checks on start (#1042)
* xftp-server: run expiration checks on start

* rename

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-01 11:29:19 +01:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Evgeny Poberezkin
44410535fd do not pass key to control port of xftp server (#1074) 2024-03-28 18:16:36 +00:00
Alexander Bondarenko
bbc9eccf4d xftp: prevent overwriting completed upload (#1063)
* xftp: prevent overwriting completed upload

* add size check for skipCommitted

* fix import

* fail on incorrect size

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-28 18:12:48 +00:00
Alexander Bondarenko
ee90ea6a69 replace base64-bytestring with base64 (#1065)
* replace base64-bytestring with base64

* minify

* use bytestring-0.10 compatible fork

PR pending...

* bump base64 fork with text compat

* move compat details to modules

* switch repo

* add back module

* cleanup

* minify

* clean imports

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-27 23:35:09 +00:00
Alexander Bondarenko
5e0123313c log control port commands, authentication for control port (#1071)
* log control port commands

* auth

* add auth to xftp, config and commands

* log missing auth

* put smp save under auth

* corrections

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-26 18:04:28 +00:00
Evgeny Poberezkin
2712fbc711 add key to XFTP server control port command (#1070)
* add key to XFTP server control port command

* name
2024-03-26 09:38:28 +00:00
Alexander Bondarenko
9eacae5492 xftp: remove duplicate check in receiveServerFile (#1057) 2024-03-20 15:50:38 +00:00
Evgeny Poberezkin
46e49fa823 xftp: return OK on duplicate upload attempt 2024-03-19 13:52:07 +00:00
Alexander Bondarenko
db3bddecca xftp-server: fix receiveServerFile (#1048)
* xftp-server: fix receiveServerFile

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-19 13:34:21 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Evgeny Poberezkin
7f7a77c4eb SMP protocol: optimize batching transactions, remove Builder (#961)
* remove Builder

* fewer chunks

* remove lazy bytestrings

* optimize

* pad
2024-01-14 20:42:47 +00:00
Evgeny Poberezkin
ca527b4d6c use Builder for batching commands (#953)
* use Builder for batching commands

* refactor

* refactor

* increase timeout in test

* do not materialize builder

* use concat

* debug test

* debug2

* delays

* more delay, less worker delay

* remove logs

* reduce delay

* tests

* sized builder

* builder

* comment

* remove unsafe

* style

* remove fromIntegral
2024-01-08 22:54:27 +00:00
Alexander Bondarenko
34056b9d7b xftp-server: add control port (#951)
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-05 17:36:19 +00:00
Alexander Bondarenko
b4495bb4f0 xftp-server: fix stats (#944)
* xftp-server: fix stats

* count file expirations

* rename

* simplify

* check files size

* store and fix fileSize check

* do

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-01-05 11:30:27 +00:00
Alexander Bondarenko
7eb7bd5e81 xftp-server: add inactiveClientExpiration (#936)
* xftp-server: add inactiveClientExpiration

* fix test config

* add test

* add xftpPing

* switch to PCEUnexpectedResponse

* remove watchdog when server quits

* rename, loop

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-27 20:31:19 +00:00
Evgeny Poberezkin
13a60d1d39 use ChaChaDRG as the source of randomness (#920)
* use ChaChaDRG as the source of randomness

* remove functions using entropy directly

* comment
2023-12-21 00:12:08 +00:00