126 Commits

Author SHA1 Message Date
Evgeny
b7a9542213 smp server: short links and owners for channels (#1506)
* smp server: short links and owners for channels

* types

* support mutliple rcv keys

* fix down migration, test/create server schema dump

* reduce schema dump

* parameterize type for link data by connection type

* return full connection link data

* test version

* change short link encoding

* test: print pg_dump output

* server pages, link encoding

* fix connection request when queue data and sender ID are created for old servers

* test, change pattern

* ci: install postgresql tools in runner (#1507)

* ci: install postgresql tools in runner

* ci: docker shell abort on error

* fix pattern for ghc 8.10.7

* patch ConnReqUriData SMP encoding to preserve queue mode after decoding

* test for RKEY

* fix/test store log with RKEY

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-04-10 19:09:47 +01:00
Evgeny
56bec06856 smp protocol: remove creating notifications from NEW command, new ideas for notifications (#1500) 2025-03-30 19:56:16 +01:00
Evgeny
04cbed90fb agent: set/update and delete contact short link data (#1499)
* agent: set/update and delete contact short link data

* delete contact link data, tests

* comments

* type signature for GHC 8.10.7
2025-03-30 11:17:25 +01:00
Evgeny Poberezkin
1dd677eec2 Merge branch 'master' into short-links 2025-03-28 19:50:18 +00:00
Evgeny
7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
Evgeny
b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny
019a32a623 smp server: optionally append store log with postgres storage (without loading and compacting, for debugging during migration) (#1480)
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)

* refactor

* remove comment
2025-03-13 10:25:56 +00:00
Evgeny
d44f09d111 tests: pass AStoreType to agent test as a parameter (#1479)
* tests: run agent tests with PostgreSQL SMP servers

* agent tests with postgres database

* enable tests

* fix store log tests

* fix test
2025-03-12 21:47:44 +00:00
Evgeny Poberezkin
e4b9aa9746 Merge branch 'master' into postgres 2025-03-09 21:58:39 +00:00
sh
36f5539b9a ci: introduce reproducible builds (#1476)
* ci: introduce reproducible builds

* ci: return 20.04

* smp server: increase timing test threshold

* ci: test outside docker

* ci: fix test step

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-03-07 14:15:18 +00:00
Evgeny
172540984c agent, smp server: PostgreSQL connection pool (#1468)
* agent, smp server: PostgreSQL connection pool

* simplify, create all connections before start

* rename

* remove imports
2025-02-25 15:33:54 +00:00
Evgeny
4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny
6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
Evgeny
3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
spaced4ndy
e557dc86e7 Merge branch 'master' into postgres 2024-12-20 17:18:54 +04:00
spaced4ndy
cf66aadc20 postgres: store implementation, conditional compilation (#1421)
* postgres: implementation wip

* to from field

* agent store compiles

* methods

* create store

* tests wip

* migration tests pass

* tests compile

* fix tests

* tests wip

* bool int

* tests wip

* tests wip

* more boolint

* more fixes

* more fields pass

* more fixes

* binary

* instances, binary

* test passes

* remove todos, more tests pass

* fix conflict

* fix bool

* fix sequence breaking

* fix insertedRowId

* skip ratchet re-synchronization tests

* after test

* file tests

* after test

* rename

* remove comment

* format

* remove unused

* suppress notices

* fixes

* move

* fix

* instance

* instance2

* fix

* instances

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-20 15:54:58 +04:00
Evgeny
4d640c16aa smp server: log prometheus metrics (#1411)
* smp server: log prometheus metrics

* save metrics

* diff

* lines

* version

* do not include Prometheus into client

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* add timestamp to metrics

* remove type

* remove version

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2024-12-18 11:19:13 +00:00
Evgeny
21fbbf9106 smp server: update message counts during message expiration, increase idle interval (#1404)
* smp server: update message counts during message expiration, increase idle interval

* version

* fix

* flip results

* version
2024-11-17 09:44:40 +00:00
Evgeny
d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
Evgeny
56ca7bf30e smp server: do not log progress when server starts (#1390)
* smp server: do not log progress when server starts

* test
2024-10-26 10:33:18 +01:00
Evgeny
c9c075fd49 smp server: merge quota messages and set queue to "over quota" state after restoring, server tests with journal and memory store (#1380)
* smp: run server tests with journal and memory store

* merge quota messages, set queue to "over quota" state after restoring

* fix test
2024-10-23 09:17:23 +01:00
Evgeny
1484523164 smp server, journal store: check file size on append, validate queue state (#1377)
* smp server: journal store queue state validation

* validate state on update, refactor

* handle message parsing errors

* typo

* fix test, throw exception when file is smaller than expected

* fix

* fix test in ghc 8.10.7
2024-10-22 12:09:51 +01:00
Evgeny
0414ea59f0 smp server: journal message store (#1370)
* smp server: remove STM function from MsgStore

* polymorphic MsgStore

* jourmal storage for messages (WIP)

* more journal, test setup

* writeMsg

* test

* tryDelMsg

* delMsgQueue

* remove MsgStoreClass instance of existential wrapper for Msg stores

* store config

* extract common logic out of store instances

* add store type to config

* open journals, cache last message, tests pass

* CLI commands

* refactor import/export messages

* cli commands to import/export journal message store

* export journal without draining, import/export tests

* journal command

* import/export progress

* better progress info

* only log queue state once when importing

* logs

* handle IO errors in journal store, return as STORE error

* recover from state file errors

* fix message files after crash

* fix messages folder
2024-10-21 11:50:30 +01:00
Evgeny
8870442830 smp server: reject SKEY with different key earlier (at verification step), add tests (#1366) 2024-10-11 12:13:52 +01:00
Evgeny Poberezkin
61b2b9df1a Re-commit: smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
This reverts commit 0ba3e69872.
2024-10-02 12:23:46 +01:00
Evgeny Poberezkin
0ba3e69872 Revert "smp server: persist notifications to avoid losing them when ntf server is offline (#1336)"
This reverts commit 0739f7b702.
2024-10-02 09:49:43 +01:00
Evgeny
0739f7b702 smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
* ntf server: types for storing notifications

* fix tests

* remove comments

* batch NMSGs test

* fix test

* thread to expire notifications

* persist notifications on restart

* optimize

* refactor

* remove reverse

* control port stats
2024-09-29 18:50:16 +01:00
Evgeny
7c25b3b1e0 smp protocol: send DELD when subscribed queue is deleted (#1312)
* smp protocol: send DELD when subscribed queue is deleted

* fix, test

* refactor

* send DELD event only if the client supports it (version 10); send END otherwise

* fix test

* notify on notifier rotation

* increase test delays
2024-09-11 13:16:51 +01:00
Evgeny
137ebc1cad servers: reduce memory used for period stats (#1298) 2024-09-02 23:12:08 +01:00
Evgeny
ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny
655e7ad7d5 smp server: get message queue faster, avoiding STM contention if queue exists, split transaction for notification delivery (#1289)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* smp server: get message queue faster, avoiding STM contention if queue exists

* IORef for counter

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* version

* remove IORef

* split notification delivery transations

* revert version
2024-08-30 11:53:22 +01:00
Evgeny
2e7e476f81 smp server: remove "expensive" stats (#1285) 2024-08-28 18:20:44 +01:00
Evgeny
16cf5c8628 smp server: stats for END events and for SUB/DEL event batches (#1281)
* smp server: count queued and sent END events

* fix

* shadowing

* stats for batches

* fix
2024-08-26 14:58:18 +01:00
Evgeny Poberezkin
5fa3c149e9 smp server: fix server-info, additional stats, allow expiring inactive clients which have prohibit subscriptions only (iOS NSE clients) (#1237)
* smp server: fix server-info

* fix

* faster saving messages

* remove comment

* move ProhibitSub out of TVar

* subscription stats

* stabilize test

* order

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* more notification stats

* count ntf stats

* update server-info

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-07-25 16:06:40 +01:00
Evgeny Poberezkin
e59a098e66 smp server: remove subscriptions from the clients when queues are deleted (#1234)
* smp server: remove subscriptions from the clients when queues are deleted

* remove ntf subscriptions, update stats

* add GET stats
2024-07-18 10:59:48 +01:00
Evgeny Poberezkin
21abc5cabe smp server: reduce the number of threads by delivering message to subscription when it is sent (#1222)
* smp server: reduce the number of threads by delivering message to subscription when it is sent

* test delay

* test delay
2024-07-08 23:12:01 +01:00
Evgeny Poberezkin
9ee684b0f4 rfc: faster handshake protocol (#1203)
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
2024-06-30 08:36:24 +01:00
Evgeny Poberezkin
388d77b61a ntf stats on SMP server (#1197)
* log undelivered notifications

* type

* add counters and encoding

* rename

* diff

* style

* style2

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-06-15 20:05:45 -07:00
Evgeny Poberezkin
e55ec07fe2 server: log stats for QUOTA and other errors (#1177)
* server: log stats for QUOTA errors

* fix test

* more stats

* remove duplicate column
2024-05-28 15:32:41 +01:00
Alexander Bondarenko
f89d715a99 smp server: add proxy stats (#1157)
* smp-server: add proxy counters

* count simplex.im messages

* update

* fix

* get own servers from INI

* remove export

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-20 17:07:33 +01:00
Evgeny Poberezkin
2d1609f222 update envelope sizes for proxied messages, remove unnecessary proxy-relay encryption padding (#1107)
* update envelope sizes for proxied messages

* remove unnecessary padding from proxy-relay encryption
2024-04-19 20:24:25 +01:00
Evgeny Poberezkin
ad4b5b6b71 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-15 13:47:48 +01:00
Alexander Bondarenko
875ddd80d6 SMP proxy: protocol (#954)
* WIP: proxy-related types

* test plan

* buildable with stubs

* add auth test

* update protocol

* fix

* update rfc

* update protocol/types

* disable test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-12 18:04:29 +01:00
Evgeny Poberezkin
1219446996 dispose agent instances, fix tests, (#1089)
* dispose agent instances in tests

* fix quota test

* tests: fix tests with -threaded (#1088)

* fix some tests

* match RTS opts with apps

* less verbose rts stats

* enable sqlite extended error codes

* clean up

* unfocus

* remove extendedErrorCode

It's actually setExtendedResultCodes, which isn't yet available.

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* fix switch test

* fix

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-04-10 19:34:02 +01:00
Evgeny Poberezkin
84b8c8417b revert base64 change (#1081) 2024-04-03 09:27:36 +01:00
Evgeny Poberezkin
6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
Alexander Bondarenko
ee90ea6a69 replace base64-bytestring with base64 (#1065)
* replace base64-bytestring with base64

* minify

* use bytestring-0.10 compatible fork

PR pending...

* bump base64 fork with text compat

* move compat details to modules

* switch repo

* add back module

* cleanup

* minify

* clean imports

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-03-27 23:35:09 +00:00
Evgeny Poberezkin
ba1bfaa5aa Merge branch 'master' into pq 2024-03-04 20:13:01 +00:00
Evgeny Poberezkin
dd2bd11584 parameterize version scopes with phantom types (#1026)
* parameterize version scopes with phantom types

* move Version to another module

* parens
2024-03-04 19:06:51 +00:00
Evgeny Poberezkin
e06e22328f agent: quantum-resistant double ratchet encryption (#939)
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-03-03 19:40:49 +00:00