dandri/simplexmq
1
0 Fork 0
mirror of https://github.com/simplex-chat/simplexmq.git synced 2026-11-06 14:48:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,498 Commits 138 Branches 186 Tags
ae8e1c5e9aa3155907f1bd075e9c69af5fce2bee
Commit Graph
77 Commits
Author SHA1 Message Date
spaced4ndyandEvgeny Poberezkin ae8e1c5e9a agent: servers stats improvements, fixes (#1208) 
* agent: reset stats startedAt time in memory

* getAgentSubsSummary

* change sub counting

* ack statistics

* add import

* instance

* Revert "instance"

This reverts commit 1f63740d56.

* Revert "add import"

This reverts commit ef72df8014.

* modify sub counting

* modify conn creation counting

* use int64

* file size stats

* remove import

* ack err counting

* conn del stats

* format

* new data

* add data

* toKB

* restore connCompleted

* use Int for counts

* use rq from scope

* remove getAgentSubsSummary

* fix connCompleted

* fix

* revert disabling stats

* use srv from scope

* combine ack stats

* modify

* comment

* count subs

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
 2024-07-03 12:36:15 +01:00
Evgeny Poberezkin 5c839b40c9 Merge branch 'master' into fast-connection 2024-07-01 21:14:32 +01:00
spaced4ndy a50e2e74a5 agent: disable saving stats (#1214) 
* agent: disable stat saving

* disable migration

* schema

* disable restore
 2024-07-01 21:06:38 +01:00
Evgeny Poberezkin 9ee684b0f4 rfc: faster handshake protocol (#1203) 
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
 2024-06-30 08:36:24 +01:00
spaced4ndyandEvgeny Poberezkin c788692687 agent: servers summary types, api (#1202) 
* agent: servers summary types, api [wip]

* encoding

* export

* Revert "export"

This reverts commit cd9f315fe8.

* comment

* rename

* simplify types

* uncomment

* comment

* rework

* comment, exports

* save, restore stats wip

* remove

* rename

* save stats periodically

* sigint, sigterm experiments

* corrections

* remove some proxy stats

* increase stat

* proposed stats

* fields

* Revert "sigint, sigterm experiments"

This reverts commit f876fbd418.

* wip

* retries -> attempts

* errs

* fix

* other errs

* more stat tracking

* sub stats

* remove xftp successes stats

* xftp stats tracking

* revert

* revert

* refactor

* remove imports

* comment

* Revert "refactor"

This reverts commit 26c368d82a.

* Revert "revert"

This reverts commit 4c9e3753b5.

* Revert "revert"

This reverts commit 6f65644053.

* todos

* persistence

* rename, fix

* config

* comment

* add started at to summary

* delete stats on user deletion

* reset api

* move

* getAgentServersSummary collect state logic

* corrections

* corrections

* remove

* rework

* decrease contention

* update

* more stats

* count sentProxied

* count subs

* remove unused

* comment

* remove comment

* comment

* export

* refactor

* cleanup

* intervals

* refactor

* refactor2

* refactor3

* refactor4

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2024-06-25 09:42:59 +04:00
spaced4ndy 2e5433676e xftp: check proxy before downloading from unknown server (#1102) 
* xftp: check proxy before downloading from unknown server

* corrections

* remove import

* tests

* comment

* tests

* don't wrap into internal error

* fix tests
 2024-05-20 14:14:04 +01:00
Evgeny Poberezkin 4ffb6a348a pqdr: use different newtypes for supporting and enabling PQ encryption in connections (#1031) 
* pqdr: use different newtypes for supporting and enabling PQ encryption in connections

* rename field, fix test

* refactor
 2024-03-06 21:28:03 +00:00
Evgeny Poberezkinandspaced4ndy e06e22328f agent: quantum-resistant double ratchet encryption (#939) 
* doc

* diff

* ratchet header

* types

* ratchet step with PQ KEM, message header with KEM

* comment

* update types, remove Eq instances, store KEM keys to database

* pqx3dh

* PQ double ratchet test

* pqdr tests pass

* fix most tests

* refactor

* allow KEM proposals from both sides

* test names

* agent API parameters to use PQ KEM

* initialize ratchet state for enabling KEM

* fix/test KEM state machine to support disabling/enabling via messages

* more tests

* diff

* diff2

* refactor

* refactor

* refactor

* refactor

* remove Maybe

* rename

* add PQ encryption status to CON, MID and MSG events and sendMessage API results

* different PQ parameter when creating connection

* rename/reorganize types for PQ encryption modes

* rename

* fix testWaitDeliveryTimeout

* rename

* rename2

* ghc8107

* rename

* increase timeouts for concurrent send/receive test

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
 2024-03-03 19:40:49 +00:00
spaced4ndyandEvgeny Poberezkin 294d7ec8dd agent: delay connection deletion to finish delivery of pending messages (#1015) 
* agent: delay connection deletion to finish delivery of pending messages (wip)

* fixes, test

* notify, test

* add tests

* comment

* add test

* timeout

* test timeout

* up

* more tests

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
 2024-02-29 18:08:58 +00:00
Alexander Bondarenko 194a7bb58e agent: remove redirect entries when deleting rcv files (#995) 
* agent: remove redirect entries when deleting rcv files

* update schema dump

* use rcv_file primary key for redirect lookup

* allow multiple redirects
 2024-02-16 19:49:50 +04:00
spaced4ndy 004597c764 agent: add index to file redirect migration (#988) 2024-02-14 18:47:39 +04:00
Alexander BondarenkoandEvgeny Poberezkin 2f7a288280 xftp: add sending and receiving via URI-encoded redirects (#968) 
* xftp: add URI encoding for FileDescription

* tweak URI

* allow smaller blocks

* draft xftpReceiveFileFollow' and xftpSendFilePublic'

* add sending with redirect

* allow 64k chunks

* add migrations with redirect fields

* add test case

* fix deadlock

* revert CLI code

* WIP: working send/receive via URI

* fix field ambiguity

* cleanup

* update agent db schema

* update minimal chunk size

* add rfc

* apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add createRcvFileRedirect

* extract Simplex.Messaging.ServiceScheme and reuse for files

* update db schema

* check size/digest on receive complete

* cleanup

* use SIZE/DIGEST errors for redirects too

* split digest/size errors from redirect checks

* fix redirect error encoding

* rename RedirectMeta to RedirectFileInfo

* use query encoding for file URI

* group maybe fields under RcvFileRedirect

* add extras field

* update rfc

* add extras encoding and no-redirect tests

* fix toStrict for old ghc

* extra client data in file descr URI

* remove decoded yaml file

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
 2024-02-13 14:08:49 +00:00
Evgeny Poberezkinandspaced4ndy fd4eeb36db agent: optimize expired messages query (#976) 
* agent: optimize expired messages query

* schema

* fix query

* fix

* typo

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix

* refactor

* comment

* refactor2

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
 2024-01-22 14:04:57 +00:00
Evgeny Poberezkin b547f34cc0 Revert "Revert "agent: do not create user record in new databases (#957)" (#963)" (#964) 
* Revert "Revert "agent: do not create user record in new databases (#957)" (#963)"

This reverts commit aee9088417.

* tests

* remove logging
 2024-01-17 11:01:04 +00:00
Evgeny Poberezkin aee9088417 Revert "agent: do not create user record in new databases (#957)" (#963) 
This reverts commit 9b9a0bd0df.
 2024-01-14 22:28:28 +00:00
Evgeny Poberezkin 9b9a0bd0df agent: do not create user record in new databases (#957) 
* agent: do not create user record in new databases

* update
 2024-01-12 16:57:57 +00:00
Evgeny Poberezkinandspaced4ndy 7ddeca50e4 agent: mark work items failed (#931) 
* agent: mark work items failed (WIP)

* add tests, created_at

* getWorkItem for snd and rcv files

* fix

* tests

* fix

* tests

* test

* tests

* rename

* fix,refactor

* add indexes

* update schema

* do not try to get more work when resuming an existing worker

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
 2023-12-26 17:00:42 +04:00
Evgeny Poberezkin 1e15d56e92 5.5.0.1: fix migration 2023-12-23 16:06:11 +00:00
Evgeny Poberezkin 2489333c87 agent: use DB to queue async commands and messages (#929) 
* agent: use DB to queue async commands and messages

* fix message delivery, async commands dont work

* fix async commands

* remove comment

* rename

* comment

* balance agent operation

* empty lines

* balance another agent operation

* diff

* worker

* qAddress

* fix incorrect queue ID in v1

* types for queues

* fix test

* update index
 2023-12-23 12:59:53 +00:00
Evgeny Poberezkin 5dc3d739b2 agent: support encrypted local files (#837) 
* agent: support encrypted local files

* migration, update store, api

* tests, fix

* use CF.plain
 2023-08-31 22:43:58 +01:00
spaced4ndy e586bef57a agent: add indexes to improve slow queries performance (#823) 2023-08-16 10:29:09 +04:00
spaced4ndy e81f3b5529 add indexes (#809) 2023-07-22 13:20:49 +01:00
spaced4ndy 7e2b309450 migration to delete expired messages (#801 is suspected to have caused temporary traffic increase due to flood of non expired messages) (#806) 2023-07-20 16:15:33 +04:00
Evgeny Poberezkinandspaced4ndy 58cb2855d2 agent: delivery receipts (#752) 
* rfc: delivery receipts

* update doc

* update rfc

* implementation plan, types, schema

* migration, update types

* update types

* rename migration

* export MsgReceiptStatus, JSON encoding

* update rfc, schema

* correction

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* skeleton of the implementation

* more implementation (some tests fail)

* more code, 1 test fails

* fix encoding

* refactor

* refactor

* test, fix

* only send receipts in v3+, test

* flip condition

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* flip condition

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* agent version 4 required to send receipts

* fix test

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
 2023-07-13 22:33:48 +01:00
spaced4ndyandEvgeny Poberezkin 8be2505fa0 ratchet re-synchronization (#774) 
* ratchet re-synchronization rfc wip

* additions

* additions, types

* fix tests

* re-sync implementation wip

* re-sync implementation 1st rev.

* test wip

* test passes

* doc

* wording

* improve doc schema

* single agreed state

* refactor (1 state variable)

* allowed -> required

* prohibit enqueue

* enqueue

* send via multiple queues

* test with server offline

* clarify errors

* rename

* more tests

* refactor

* rename AgentRKey

* rename AM_CONN_RATCHET_KEY

* more tests

* rename

* write encoded AgentRatchetInfo to AgentRatchetKey info

* move withConnLock

* refactor qDuplex

* re-create ratchet on receiving second key

* invert condition

* refactor

* simplify

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2023-06-30 14:17:08 +04:00
spaced4ndyandEvgeny Poberezkin 011e1109e9 connection switch - track status, stop switch command (#764) 
* connection switch improvements

* stop in transaction

* refactor

* checks in transaction

* withSwitchingRQ in transaction

* withSwitchedSQ in transaction

* refactor

* refactor

* inverse some cases

* reset snd switch progress on repeat qadd

* remove unused functions

* connection info type

* correction

* SPFinalizing phase, correctly update connection in event

* refactor

* tests

* refactor switch conn lock taking, take lock in switch stop, remove ineligible test

* comment

* corrections

* corrections

* corrections

* refactor

* remove checks in transactions, remove some statuses

* remove diff

* refactor

* refactor tests

* update connection in memory

* revert diff

* corrections

* more tests

* remove statuses, change switching queue lookup

* async queue deletion

* refactor

* lock, phases

* test status

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2023-06-15 11:15:46 +04:00
spaced4ndy 6c6f22051d check previous encrypted message hashes for duplicates on decryption errors (#756) 2023-05-17 15:42:57 +04:00
spaced4ndy ce64c91d5a add indexes for reading pending files (#751) 2023-05-10 15:17:26 +04:00
spaced4ndyandEvgeny Poberezkin 8954f39425 agent: vacuum on migrations (#745) 
* agent: vacuum on migrations

* vacuum after all migrations

* style

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2023-05-04 17:26:05 +04:00
spaced4ndy 9f0b9a83d6 xftp: use chunk digest as entity id for transport session (#725) 2023-04-14 13:01:14 +04:00
spaced4ndy e346a81e46 xftp: delete snd files on servers (#722) 2023-04-13 17:46:43 +04:00
spaced4ndy d8e60ecfdb xftp: recoverable send - spike (#707) 2023-04-07 11:19:50 +04:00
Evgeny Poberezkinandspaced4ndy 6a665a0833 support down migrations (#696) 
* support down migrations

* test

* update

* add down migrations and tests

* get current migrations

* corrections

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* update tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
 2023-03-27 12:44:53 +01:00
Evgeny Poberezkin d41c2bec2a store retry intervals to resume from the same interval on restart (#693) 
* store retry intervals to resume from the same interval on restart

* add migration

* updates
 2023-03-22 08:42:56 +00:00
spaced4ndy 2bec353eae xftp: add command to delete rcv file; agent manages save path (#692) 2023-03-20 20:08:38 +04:00
spaced4ndy 9255d48b69 xftp: move delay to replica (#688) 2023-03-16 18:12:38 +04:00
spaced4ndy db120b6d2e xftp: pass save path to agent (#685) 2023-03-14 19:16:25 +04:00
spaced4ndy cf8f78f518 xftp: add random bytestring entity id for rcv files (#681) 2023-03-10 20:32:37 +04:00
spaced4ndy cc2066473c xftp: cleanup complete & error files tmp paths; remove save path before decrypting if exists (#677) 2023-03-10 14:26:34 +04:00
spaced4ndy deec963de8 xftp: agent error handling (#675) 
- chunk download doesn't loop on permanent errors
- decryption errors are considered permanent - local worker doesn't retry
- update replica retries; to do - consider use for this field, or remove it
- rcv file Error status - to prevent repeat reads of chunks for download, files for decryption; also plan to use it for filtering on cleanup
- error string saved in separate field for debugging (not part of status type)
- agent event for rcv file errors
 2023-03-09 15:32:17 +04:00
spaced4ndy 6d52150672 xftp: agent send file design wip (#674) 2023-03-08 14:30:37 +04:00
spaced4ndy a0c86158d9 xftp agent: update chunk delay (#671) 2023-03-03 20:39:15 +04:00
spaced4ndyandEvgeny Poberezkin cf147397a4 xftp: agent receive file (#637) 
* xftp: agent receive file draft

* receiveFile more stubs, changes to types, schema

* cabal file

* comments

* xftp_server_id

* schema changes, get client, local worker/action

* agent env, save file description, adjust schema

* client stubs

* download chunk wip, store, schema, types

* remove commented code

* read file description, schema, types

* check received, decrypt

* remove pure

* todo

* add XFTP to agent client

* add user id

* agent test

* tests

* rename supervisor into agent

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2023-03-03 18:50:16 +04:00
spaced4ndyandEvgeny Poberezkin c0dcf283eb server_key_hash fields (#643) 
* server_key_hash fields

* test

* refactor

* fix

* order

* use sync command in test

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
 2023-02-17 21:24:32 +00:00
Evgeny Poberezkin d4fc638478 async batch connection deletion (#617) 
* async batch connection deletion

* delete user record with connections, async connection deletion

* updates

* update query
 2023-01-21 13:41:16 +00:00
JRoberts 42df6a421d add missing fkey indexes (#610) 2023-01-17 16:07:13 +04:00
Evgeny Poberezkin 324e01300d execute asynchronous commands for correct users, include session in messages from SMP clients to agent (#608) 
* execute asynchronous commands for correct users

* pass transport session to message processing to avoid race condition

* account for server changes when determining unused servers

* enable one test

* enable all tests, remove log
 2023-01-16 11:06:19 +00:00
Evgeny PoberezkinandJRoberts f4ad3a983e support users in agent to isolate traffic of different users (#598) 
* users table, isolate traffic sessions by users or by queues

* remove extra indices

* corrections

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
 2023-01-11 17:47:20 +04:00
Evgeny Poberezkin 1d3a604988 refactor SMP client, NTF server store (#580) 
* refactor SMP client, NTF server store

* remove undefined
 2022-12-26 18:54:28 +00:00
Evgeny PoberezkinandJRoberts 8d9816809f simplify queue rotation protocol (#550) 
* simplify queue rotation protocol

* use simplified rotation protocol, update tests

* simplify schema

* delete all connection queues

* refactor

* switch notifications to the new queue, tests

* remove TODO

* refactor

* rfc correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* remove duplicate set active

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
 2022-10-31 13:33:28 +04:00