Commit Graph

107 Commits

Author SHA1 Message Date
Evgeny
b90e25a3a5 ntf server: fix repeat token registration when it became invalid (regression) (#1539) 2025-05-15 19:56:48 +01:00
Evgeny Poberezkin
cf4b9f669d agent: always enable periodic notifications 2025-05-15 15:16:44 +01:00
Evgeny
0a3d014f5d ntf server: priority db pool for notifications and client commands, concurrent resubscriptions, CLI options (#1529)
* ntf server: option to skip specific tokens when importing store log

* logs

* batch later

* subscribe in parallel, bigger queues

* ntf server: test schema and migrations

* optimize subscriptions

* log level

* resubscribe before starting

* better subscribing

* pooled concurrent resubscriptions

* 5

* async resubscribe

* typo

* priority pool

* db batch size
2025-04-30 11:40:32 +01:00
Evgeny
7d0115daec ntf server, agent: send all periodic notifications from one thread, only to old active clients or new clients with periodic notification mode (#1528)
* ntf server, agent: send all periodic notifications from one thread, only to old active clients or new clients with periodic notification mode

* send different type via subscription queues

* option to compact store log on start
2025-04-28 12:18:55 +01:00
Evgeny
f024ab1c3f ntf server: prometheus metrics (#1527)
* ntf server: save prometheus stats

* info metrics

* fix test
2025-04-27 13:25:44 +01:00
Evgeny
f4bc1f0926 ntf server: skip duplicates when importing tokens and subscriptions (#1526)
* ntf server: skip duplicates when importing tokens and subscriptions

* skip imported last notifications when no token or subscription present

* fix skipped imported notifications count

* all tests

* fix test
2025-04-26 23:37:55 +01:00
Evgeny
42dbb887f7 ntf server: allow retries when creating subscriptions, prohibit subscriptions with the same queue but another notifier key or token (#1525)
* ntf server: allow retries when creating subscriptions, prohibit subscriptions with the same queue but another notifier key or token

* sync files in the test

* refactor
2025-04-25 19:41:18 +01:00
Evgeny
850d2fa423 ntf server: PostgreSQL database storage (#1519)
* ntf server: PostgreSQL database storage

* ntf server: import/export stubs

* ntf server postgres db functions

* some notifications tests pass

* notifications tests pass

* import/export notification store logs

* fix ntf server CLI

* log in parralel

* update subscription statuses using executeMany

* fix import/export

* refactor

* fix queries

* prohibit token_id and subscription_id updates, dont update tokens and subscriptions on conflict, improve server insertion, remove duplicate tokens for import, remove subscriptions without tokens for import

* comment

* remame

* increase test delay
2025-04-25 16:47:39 +01:00
Evgeny Poberezkin
6b60f8bab6 Revert "ntf server: optimize in-memory storage (#1516)"
This reverts commit afb338a41a.
2025-04-21 21:01:24 +01:00
Evgeny
afb338a41a ntf server: optimize in-memory storage (#1516)
* ntf server: optimize in-memory storage

* test

* ntf server: fix store log parser for token status
2025-04-21 17:12:16 +01:00
Evgeny
5dbe6337ea ntf server: additional statistics, new invalid token reasons (#1451)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* ntf server: additional statistics

* swap

* version

* more stats

* test, verify invalid

* rename

* exclude test token from stats

* increase delay

* handle invalid token in retry, more reasons

* focus tests

* disable new tests in CI

* fix

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-07 12:19:11 +00:00
spaced4ndy
b633f89c1a agent: check ntf token status on registration (#1450)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* test, verify invalid

* rename

* increase delay

* disable new tests in CI

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-07 11:36:29 +00:00
Evgeny
944a22a2fb ntf server: record token invalidation reason, add date of the last token activity (#1449)
* ntf server: record token invalidation reason, add date of the last token activity

* update time

* rename

* optional

* include token ID in delivery error

* version

* protocol version

* fix, log error
2025-02-06 17:11:35 +00:00
Evgeny
d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
spaced4ndy
a8471eed5b ntf: get messages for multiple last notifications (#1378)
* ntf: get messages for multiple last notifications (#1352)

* ntf: separate get ntf conns api (#1379)

* ntf: separate get ntf conns api

* nonempty

* update

* update

* remove single get api

* fix test

* refactor

* refactor

* ntf: batch get connections (#1387)

* ntf: batch get apis

* works

* fix

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-25 15:39:41 +04:00
Evgeny
50e4abf836 ntf server: control port (#1354)
* ntf server: control port

* version .405

* control

* use own_server_domains from INI file

* fix subs by server in control port

* bigger queues

* ntf server: only print subscriptions per own server when they are > 0

* fix tests

* revert version change

* dont import listThreads in ghc 8.10.7
2024-10-06 12:33:14 +01:00
Evgeny Poberezkin
61b2b9df1a Re-commit: smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
This reverts commit 0ba3e69872.
2024-10-02 12:23:46 +01:00
Evgeny Poberezkin
0ba3e69872 Revert "smp server: persist notifications to avoid losing them when ntf server is offline (#1336)"
This reverts commit 0739f7b702.
2024-10-02 09:49:43 +01:00
spaced4ndy
9cf6c97137 ntf: batch ntf worker command processing (#1337)
* ntf: batch ntf worker command processing

* remove comment

* change batch size

* wip

* catch

* refactor

* refactor

* batch check

* refactor

* reschedule

* increase ntfSubCheckInterval

* first check interval

* check more statuses

* refactor, remove foldr'

* refactor 2

* refactor client

* refactor 3

* ntf server: improve support for batched commands (#1340)

* re-create ntf subscriptions on NTF AUTH errors

* name

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-29 18:50:38 +01:00
Evgeny
0739f7b702 smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
* ntf server: types for storing notifications

* fix tests

* remove comments

* batch NMSGs test

* fix test

* thread to expire notifications

* persist notifications on restart

* optimize

* refactor

* remove reverse

* control port stats
2024-09-29 18:50:16 +01:00
Evgeny
2a120dfe57 smp server: Allow serving HTTPS and transport on the same port (v2) (#1327)
* smp-server: Allow serving HTTPS and transport on the same port

* update rfc

* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started

* diff

* combine HTTP and SMP on the shared port

* Update to SockAddr

* Fix params and web.https parser

* Switch fork urls

* WIP: add smpServerTestStatic test

* Update warp-tls repo

* shared connection tests

* cleanup

* Add protocol tests

* rename cert file, enable both ports and web by default

* terminate with message on missing credentials

* test cert file

* client option to use port 443 as default SMP port

* use SNI in non-SMP clients

* supported

* remove TODO

* advice

* fix test build

* Add RSA-4096 check for web creds, fix test

* Remove directory listing from static app

* message

* messages

* update log tests

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-09-28 23:15:17 +01:00
Evgeny
3c18c4b66a servers: refactor TLS credentials (#1326)
* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started
2024-09-28 22:21:08 +01:00
Evgeny
7c25b3b1e0 smp protocol: send DELD when subscribed queue is deleted (#1312)
* smp protocol: send DELD when subscribed queue is deleted

* fix, test

* refactor

* send DELD event only if the client supports it (version 10); send END otherwise

* fix test

* notify on notifier rotation

* increase test delays
2024-09-11 13:16:51 +01:00
spaced4ndy
092ed088ca ntf: support for multiple messages encoding (#1305) 2024-09-09 16:03:17 +04:00
Evgeny
ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny
9596a03139 servers: reduce STM transactions (#1287)
* servers: reduce STM transactions

* switch stats and pending ENDs to IORef

* more IORef, split pending ENDs to use in one thread
2024-08-29 13:18:12 +01:00
Evgeny
325e679bc4 ntf server: reduce PING interval to 1 minute, only mark subscriptions with END if it is received by the currently active client (#1275) 2024-08-24 11:34:59 +01:00
Evgeny
7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny Poberezkin
743676421d ntf server: simplify and optimize subscriptions in server agent (#1219)
* increase queue size

* simplify

* refactor to optimize memory usage and performance

* comment

* refactor

* test delays
2024-07-07 21:17:12 +01:00
Alexander Bondarenko
9e7e0d102d smp-server: conserve resources (#1194)
* transport: force auth params, remove async wrapper

* stricter new messages

* bang more thunks

* style

* don't produce msgQuota unless requested

* strict

* refactor

* remove bangs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-24 13:15:08 +01:00
Evgeny Poberezkin
8b21f7ef2a agent: aggregate multiple expired subscription responses into a single UP event (#1160)
* agent: aggregate multiple expired subscription responses into a single UP event

* clean up

* refactor processing of expired responses

* refactor

* refactor 2

* refactor unexpectedResponse
2024-05-20 07:56:51 +01:00
Alexander Bondarenko
426f47c805 smp: use session vars for reconnecting small agent (#1152)
* smp: use session vars for reconnecting small agent

* process errors

* split session and protocol functions

* add active flag to agent

* actually invoke agent shutdown

* close proxy agent too

* restore stopping ntf subscribers
2024-05-16 19:06:27 +01:00
Evgeny Poberezkin
91cc48aabe agent: do not mark subscriptions on expired sessions as active, mark delayed subscriptions as active on the same session, do not cancel sending expired commands (#1127)
* agent: do not mark subscriptions on expired sessions as active, do mark delayed subscriptions as active on the same session, SUBOK response in the next SMP protocol version

* client: prevent sub actions from zombie sessions (#1122)

* client: prevent sub actions from zombie sessions

* error handling

* add AERR to pass background errors to client

* switch to activeClientSession

* put closeClient under activeClientSession

* rename

* remove AERR, do not skip processing

* move check and state update to one transaction

* catch extra UPs

* fix

* check queue is still pending before making it active

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* do not forward agent error

* revert not expiring sending subs

* fixes

* track subscription responses better

* add pending connection

* Revert "revert not expiring sending subs"

This reverts commit 4310a69391.

* do not expire sending commands

* rename

* fix race

* function

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-12 17:47:08 +01:00
Evgeny Poberezkin
2347b82b47 adjust timeouts and delayed response errors (#1112) 2024-04-21 22:57:07 +01:00
Evgeny Poberezkin
2f43b43225 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-17 17:46:22 +01:00
Alexander Bondarenko
a4cfcfcc85 ntf-server: fix test token handling (#1083)
* use formatted logger

* fix DeviceToken parser for apns_null

* exclude PPNull from token stats
2024-04-04 14:46:40 +01:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Alexander Bondarenko
ff1d7f2fac ntf-server: break processing transaction (#1067)
* ntf-server: break processing transaction

* make resubscribe sync

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-25 15:35:33 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
416f1b1721 smp: command authorization (#982)
* smp: command authorization

* fix encoding, most tests

* remove old tests

* authorize via crypto_box

* extract authenticator to Crypto module

* make TransmissionAuth Maybe

* rfc

* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)

* fix/add tests, add version config to "small" agent

* separate client and server versions for SMP protocol

* test batching SMP v7

* do not send session ID in each transmission

* refactor auth verification in the server, split tests

* server "warm up" fixes timing test

* uncomment SUB timing test

* comments, disable two timing tests

* rename version

* increase auth timing test failure threshold

* use different algorithms to authorize snd/rcv commands, use random correlation ID

* transport: fetch and store server certificate (#985)

* THandleParams (WIP, does not compile)

* transport: fetch and store server certificate

* smp: add getOnlinePubKey example to smpClientHandshake

* add server certs and sign authPub

* cleanup

* update

* style

* load server certs from test fixtures

* sign ntf authPubKey

* fix onServerCertificate

* increase delay before sending messages

* require certificate with key in SMP server handshake

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove dhSecret from THandle

* remove v8, merge all changes to one version

* parameterize THandle

* rfc: transmission ecnryption

* Revert "parameterize THandle"

This reverts commit 75adfc94fb.

* use batch syntax for ntf server commands

* separate encodeTransmission when there is no key

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* rename

* diff

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-02-16 11:45:54 +00:00
Evgeny Poberezkin
8ff89c19dc servers: do not keep stats file open during delays (#974) 2024-01-19 17:47:47 +00:00
Evgeny Poberezkin
7f7a77c4eb SMP protocol: optimize batching transactions, remove Builder (#961)
* remove Builder

* fewer chunks

* remove lazy bytestrings

* optimize

* pad
2024-01-14 20:42:47 +00:00
Evgeny Poberezkin
13a60d1d39 use ChaChaDRG as the source of randomness (#920)
* use ChaChaDRG as the source of randomness

* remove functions using entropy directly

* comment
2023-12-21 00:12:08 +00:00
Alexander Bondarenko
7c27357eb2 server: stop inactive clients without subscriptions, additional monitoring via control port (#901)
* fix thread labelling points

* add subscription thread labels

* add explicit sections for sockets/sessions/disconnects

* adjust label threads wrt. next blocking op

* WIP: emit event on server finish

So the event log wouldn't be cut short giving
false impression of unterminated sessions.

* add handshake timeout

* hack server socket stats

* trace closeTLS exceptions

* fix build

* clean up disconnectTransport

* make disconnectTransport terminating

Thus, participating in racyAny_ around it.

* trach snd/rcv time separately, log clients as CSV

* add direct client counting/stats

* add csv dump for socket threads

* guard socket threads dump from older GHC

* cut socket thread listing into socket-threads command

* store client creation time for stats

* wrap tls setup in timeout and labels

* server: expire clients without subscriptions

* fix check for subscriptions

* remove excess tracing

* move hardcoded timeouts to configs

* cleanup

* fix tests

* wrap direct queue waits in timeout

* rewrite old TCP wrappers with their SocketState variants

* add testInactiveWithSubs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-19 15:50:45 +00:00
Evgeny Poberezkin
cf8b9c12ff remove test dependencies (#861)
* remove test dependencies

* .hlint.yaml
2023-10-22 10:50:36 +01:00
Evgeny Poberezkin
9d12d76078 add fourmolu (#868)
* add fourmolu

* linebreak

* simplify

* comment
2023-10-22 09:20:14 +01:00
Evgeny Poberezkin
57d08c0e36 Merge branch 'master' into master-ghc9 2023-09-17 22:30:46 +01:00
Evgeny Poberezkin
0b488ceb66 ntf server: faster restoring from store log (#846)
* ntf server: faster restoring from store log

* only log sub status when changed
2023-09-17 15:16:24 +01:00
Moritz Angermann
002f36dde0 5.4.0.0: use ghc 9.6.2 (#791)
* Add 9.6 compat

* compile with GHC9.6.2: dependencies, imports, code

* refactor typeclasses

* refactor record dot

* update cabal version

* update github actions

* update direct-sqlcipher

* 5.4.0.0

* update cabal.project

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-08-24 20:22:02 +01:00
Evgeny Poberezkin
4c6c436e7f resubscribe in batches in Client.Agent (#798)
* resubscribe in batches in Client.Agent

* fix test
2023-07-12 07:35:38 +01:00