Commit Graph

184 Commits

Author SHA1 Message Date
Evgeny 21eee2b548 smp servers: separately account for subscribed clients, to disconnect those without subscriptions more effectively (and to be able to deliver notifications to subscribed ntf servers) (#1339) 2024-09-28 19:59:53 +01:00
Evgeny 7c25b3b1e0 smp protocol: send DELD when subscribed queue is deleted (#1312)
* smp protocol: send DELD when subscribed queue is deleted

* fix, test

* refactor

* send DELD event only if the client supports it (version 10); send END otherwise

* fix test

* notify on notifier rotation

* increase test delays
2024-09-11 13:16:51 +01:00
Evgeny 990dcec348 smp server: add created/updated/used date to queues to manage expiration (#1306)
* smp server: add created/updated/used date to queues to manage expiration, all: make Map updates strict in value

* remove strict

* remove time precision

* diff

* style

* only update when time changed
2024-09-09 14:53:11 +01:00
Evgeny a9e8d02593 server: bind control port server only to 127.0.0.1 for better security (in case of firewall misconfuguration) (#1280) 2024-09-05 13:48:09 +01:00
Evgeny e86338d555 smp server: fewer map updates on re-subscriptions (#1297)
* smp server: fewer map updates on re-subscriptions

* temp version

* replace Client with ClientId in queues

* version

* version

* comments

* reduce threads when sending ENDs

* revert version
2024-09-05 13:25:41 +01:00
Evgeny 137ebc1cad servers: reduce memory used for period stats (#1298) 2024-09-02 23:12:08 +01:00
Evgeny d84a49b85a smp server: split and reduce STM transactions (#1294) 2024-09-02 17:06:31 +01:00
Evgeny ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny 655e7ad7d5 smp server: get message queue faster, avoiding STM contention if queue exists, split transaction for notification delivery (#1289)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* smp server: get message queue faster, avoiding STM contention if queue exists

* IORef for counter

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* version

* remove IORef

* split notification delivery transations

* revert version
2024-08-30 11:53:22 +01:00
Evgeny 9596a03139 servers: reduce STM transactions (#1287)
* servers: reduce STM transactions

* switch stats and pending ENDs to IORef

* more IORef, split pending ENDs to use in one thread
2024-08-29 13:18:12 +01:00
Evgeny 2e7e476f81 smp server: remove "expensive" stats (#1285) 2024-08-28 18:20:44 +01:00
Evgeny 2c9ad74599 smp server: batch END responses when subscribed client switches (version 2) (#1283)
* fix incorrect entity ID for notification subscription END when queue is deleted

* throttle sending ENDs

* fix stats

* clean up
2024-08-28 10:14:41 +01:00
Evgeny Poberezkin aa60bd6770 use strict Maps, fix stats for sent END batches 2024-08-26 19:35:57 +01:00
Evgeny 16cf5c8628 smp server: stats for END events and for SUB/DEL event batches (#1281)
* smp server: count queued and sent END events

* fix

* shadowing

* stats for batches

* fix
2024-08-26 14:58:18 +01:00
Evgeny ac930dff30 smp server: possible race when creating client that might lead to memory leak (#1260) 2024-08-20 12:25:58 +01:00
Evgeny 7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny a017047c52 smp server: fix race when client is marked as subscribed after it is disconnected, preventing its GC (#1250)
* smp server: fix race when client is marked as subscribed after it is disconnected, preventing its GC

* refactor
2024-08-06 08:17:38 +01:00
Evgeny Poberezkin 5fa3c149e9 smp server: fix server-info, additional stats, allow expiring inactive clients which have prohibit subscriptions only (iOS NSE clients) (#1237)
* smp server: fix server-info

* fix

* faster saving messages

* remove comment

* move ProhibitSub out of TVar

* subscription stats

* stabilize test

* order

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* more notification stats

* count ntf stats

* update server-info

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-07-25 16:06:40 +01:00
Evgeny Poberezkin 8423c636a8 rename queueId to entId (can be message queue, proxied session and data blob ID to be added) (#1240) 2024-07-22 12:01:38 +01:00
Evgeny Poberezkin cc5732f41f smp server: stats for commands retrieving messages when notifications are received (#1236)
* more stats

* fix stats
2024-07-20 14:10:10 +01:00
Evgeny Poberezkin e59a098e66 smp server: remove subscriptions from the clients when queues are deleted (#1234)
* smp server: remove subscriptions from the clients when queues are deleted

* remove ntf subscriptions, update stats

* add GET stats
2024-07-18 10:59:48 +01:00
Evgeny Poberezkin 492d2f86bc smp server: additional control port commands to monitor server state (#1228)
* smp server: additional control port commands to monitor server state

* fix

* space
2024-07-13 22:34:10 +01:00
Evgeny Poberezkin 26979ff6b5 smp server: simplify client subscriptions (#1223) 2024-07-09 08:36:03 +01:00
Evgeny Poberezkin 21abc5cabe smp server: reduce the number of threads by delivering message to subscription when it is sent (#1222)
* smp server: reduce the number of threads by delivering message to subscription when it is sent

* test delay

* test delay
2024-07-08 23:12:01 +01:00
Evgeny Poberezkin 743676421d ntf server: simplify and optimize subscriptions in server agent (#1219)
* increase queue size

* simplify

* refactor to optimize memory usage and performance

* comment

* refactor

* test delays
2024-07-07 21:17:12 +01:00
Evgeny Poberezkin a99ce6122c secure queue by sender via proxy (proxy SKEY command) (#1210)
* client: secure queue by sender via proxy (proxy SKEY command)

* agent and server: proxy SKEY command
2024-06-30 16:20:54 +01:00
Evgeny Poberezkin 9ee684b0f4 rfc: faster handshake protocol (#1203)
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
2024-06-30 08:36:24 +01:00
Alexander Bondarenko 9e7e0d102d smp-server: conserve resources (#1194)
* transport: force auth params, remove async wrapper

* stricter new messages

* bang more thunks

* style

* don't produce msgQuota unless requested

* strict

* refactor

* remove bangs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-24 13:15:08 +01:00
Evgeny Poberezkin 388d77b61a ntf stats on SMP server (#1197)
* log undelivered notifications

* type

* add counters and encoding

* rename

* diff

* style

* style2

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-06-15 20:05:45 -07:00
Alexander Bondarenko 6597f6f0ed tests: add proxy connection refused test (#1200) 2024-06-15 19:27:58 -07:00
Evgeny Poberezkin 3a3a84c58c server: log proxy connection errors 2024-05-31 12:20:29 +01:00
Evgeny Poberezkin 88f1b727e0 SMP protocol extension to debug subscribed SMP queues (#1181)
* SMP protocol extension to debug subscribed SMP queues

* fix, test

* corrections

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* increase delays

* increase timeout

* delay

* delay

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-05-30 18:49:43 +01:00
Evgeny Poberezkin e55ec07fe2 server: log stats for QUOTA and other errors (#1177)
* server: log stats for QUOTA errors

* fix test

* more stats

* remove duplicate column
2024-05-28 15:32:41 +01:00
Evgeny Poberezkin 4a96dbf871 server: preprocess proxy commands that will not be connecting to network to reduce concurrency, do not wait for destination relay responses before processing the next command (#1174)
* server: preprocess proxy commands that will not be connecting to network to reduce concurrency

* implementation

* tests

* increase proxy client concurrency

* simplify

* refactor

* refactor2

* rename

* refactor3

* fix 8.10.7
2024-05-28 09:38:47 +01:00
Alexander Bondarenko f50fa5c60b smp-server: limit concurrency in proxy command processing (#1162)
* smp: put client proxy command processing threads under a shared semaphore

* add LIMITS.max_proc_threads to server config

* rename to PROXY.client_concurrency

* retry on strictly greater than max concurrency

* set default to 16

* rename

* fix limit

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-21 12:53:33 +01:00
Alexander Bondarenko f89d715a99 smp server: add proxy stats (#1157)
* smp-server: add proxy counters

* count simplex.im messages

* update

* fix

* get own servers from INI

* remove export

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-20 17:07:33 +01:00
Alexander Bondarenko 426f47c805 smp: use session vars for reconnecting small agent (#1152)
* smp: use session vars for reconnecting small agent

* process errors

* split session and protocol functions

* add active flag to agent

* actually invoke agent shutdown

* close proxy agent too

* restore stopping ntf subscribers
2024-05-16 19:06:27 +01:00
Evgeny Poberezkin c4c983348f Merge branch 'master' into proxy 2024-05-13 14:13:52 +01:00
Evgeny Poberezkin 91cc48aabe agent: do not mark subscriptions on expired sessions as active, mark delayed subscriptions as active on the same session, do not cancel sending expired commands (#1127)
* agent: do not mark subscriptions on expired sessions as active, do mark delayed subscriptions as active on the same session, SUBOK response in the next SMP protocol version

* client: prevent sub actions from zombie sessions (#1122)

* client: prevent sub actions from zombie sessions

* error handling

* add AERR to pass background errors to client

* switch to activeClientSession

* put closeClient under activeClientSession

* rename

* remove AERR, do not skip processing

* move check and state update to one transaction

* catch extra UPs

* fix

* check queue is still pending before making it active

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* do not forward agent error

* revert not expiring sending subs

* fixes

* track subscription responses better

* add pending connection

* Revert "revert not expiring sending subs"

This reverts commit 4310a69391.

* do not expire sending commands

* rename

* fix race

* function

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-12 17:47:08 +01:00
Evgeny Poberezkin 8516b0dd5b proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions (#1144)
* proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions

* rename

* inline

* comment

* use correct server version when encoding forwarded commands
2024-05-11 17:11:28 +01:00
Evgeny Poberezkin 9f72b249b4 Merge branch 'master' into proxy 2024-05-11 10:25:35 +01:00
Evgeny Poberezkin 727fd8b8f5 server: more efficient response to batched subscriptions (#1141)
* server: more efficient response to batched subscriptions

* add sndMsgQ for interleaving messages with replies

* remove redundant liftIO

* refactor

* refactor2

* rename

* fix

* diff

* remove comment

* remove comment

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-10 22:19:11 +01:00
Evgeny Poberezkin 5cafd9d5c4 server: more efficient responses to batch subscriptions (#1137)
* server: more efficient responses to batch subscriptions

* comments

* comment

* enable tests

* LogError
2024-05-09 09:20:57 +01:00
Evgeny Poberezkin a70f492f4d proxy: fallback to direct connection if destination relay does not support proxy protocol (#1132)
* proxy: fallback to direct connection if destination relay does not support proxy protocol

* move version to TransportError, refactor
2024-05-07 13:37:40 +01:00
Alexander Bondarenko c85f6a2f0e proxy: reporting errors (#1108)
* smp-proxy: iron out errors

* treat proxy timeouts as temporary

* update errors

* proxy errors (missing encoding)

* update

* enable tests

* update

* update

* fix

* fix

* simplify

* test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-06 22:00:42 +01:00
Evgeny Poberezkin 2d1609f222 update envelope sizes for proxied messages, remove unnecessary proxy-relay encryption padding (#1107)
* update envelope sizes for proxied messages

* remove unnecessary padding from proxy-relay encryption
2024-04-19 20:24:25 +01:00
Alexander Bondarenko 58ede38bf4 put smp errors into proxy wrappers (#1103)
* put smp errors into proxy wrappers

* use substring in PROXY UNEXPECTED error

* fix encoding

* revert String encoding, discard invalid errors in QC

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-19 14:58:15 +01:00
Evgeny Poberezkin 7712070cb3 Merge branch 'master' into proxy 2024-04-18 23:33:06 +01:00
Evgeny Poberezkin a3b229f668 SMP proxy: low level client and server implementation (#1096)
* SMP proxy: low level client and server implementation

* SMP proxy: server implementation (#1098)

* wip

* PRXY command

* progress

* SMP Proxy: client-level implementation (#1101)

* buildable

* encode messages

* update pkey

* fix queue types

* wrap SEND in proxy lookup

* WIP proxy client

* WIP

* post-rebase fixes

* encode something with something

* cleanup

* update

* fix nonce/corrId in batchingTests

* WIP: dig into createSMPProxySession

* agent

* test progress

* pass the test

* parameterize transport handle with transport peer to include server certificate (#1100)

* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>

* cleanup

* add 2-server test

* remove subsumed test

* checkCredentials for BrokerMsg

* skip batching tests

* remove userId param

* remove agent changes

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* remove unused type

* icrease test timeout

* reduce transport block

* envelope sizes

* don't fork unless have proxied commands to process

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-18 22:35:17 +01:00
Evgeny Poberezkin 2f43b43225 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-17 17:46:22 +01:00