Commit Graph

73 Commits

Author SHA1 Message Date
Evgeny Poberezkin
fda1284ae4 agent: support closing and re-opening store (#855)
* agent: support closing and re-opening store

* add closed flag, tests
2023-09-27 12:50:24 +01:00
Moritz Angermann
002f36dde0 5.4.0.0: use ghc 9.6.2 (#791)
* Add 9.6 compat

* compile with GHC9.6.2: dependencies, imports, code

* refactor typeclasses

* refactor record dot

* update cabal version

* update github actions

* update direct-sqlcipher

* 5.4.0.0

* update cabal.project

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-08-24 20:22:02 +01:00
Evgeny Poberezkin
e2065ab352 agent: track slow SQL queries (#822)
* agent: track slow SQL queries

* add executeMany

* reduce threshold for slow queries to 50ms
2023-08-12 18:18:10 +01:00
spaced4ndy
8be2505fa0 ratchet re-synchronization (#774)
* ratchet re-synchronization rfc wip

* additions

* additions, types

* fix tests

* re-sync implementation wip

* re-sync implementation 1st rev.

* test wip

* test passes

* doc

* wording

* improve doc schema

* single agreed state

* refactor (1 state variable)

* allowed -> required

* prohibit enqueue

* enqueue

* send via multiple queues

* test with server offline

* clarify errors

* rename

* more tests

* refactor

* rename AgentRKey

* rename AM_CONN_RATCHET_KEY

* more tests

* rename

* write encoded AgentRatchetInfo to AgentRatchetKey info

* move withConnLock

* refactor qDuplex

* re-create ratchet on receiving second key

* invert condition

* refactor

* simplify

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-06-30 14:17:08 +04:00
spaced4ndy
011e1109e9 connection switch - track status, stop switch command (#764)
* connection switch improvements

* stop in transaction

* refactor

* checks in transaction

* withSwitchingRQ in transaction

* withSwitchedSQ in transaction

* refactor

* refactor

* inverse some cases

* reset snd switch progress on repeat qadd

* remove unused functions

* connection info type

* correction

* SPFinalizing phase, correctly update connection in event

* refactor

* tests

* refactor switch conn lock taking, take lock in switch stop, remove ineligible test

* comment

* corrections

* corrections

* corrections

* refactor

* remove checks in transactions, remove some statuses

* remove diff

* refactor

* refactor tests

* update connection in memory

* revert diff

* corrections

* more tests

* remove statuses, change switching queue lookup

* async queue deletion

* refactor

* lock, phases

* test status

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-06-15 11:15:46 +04:00
spaced4ndy
6c6f22051d check previous encrypted message hashes for duplicates on decryption errors (#756) 2023-05-17 15:42:57 +04:00
Evgeny Poberezkin
6a665a0833 support down migrations (#696)
* support down migrations

* test

* update

* add down migrations and tests

* get current migrations

* corrections

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* update tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2023-03-27 12:44:53 +01:00
Evgeny Poberezkin
bbf9e28696 xftp: add server to client encryption layer with cryptobox (#646)
* xftp: add server to client encryption layer with cryptobox

* workarounds for http2

* update http2 commit

* remove TODOs
2023-02-22 09:03:18 +00:00
Evgeny Poberezkin
d4fc638478 async batch connection deletion (#617)
* async batch connection deletion

* delete user record with connections, async connection deletion

* updates

* update query
2023-01-21 13:41:16 +00:00
Evgeny Poberezkin
f4ad3a983e support users in agent to isolate traffic of different users (#598)
* users table, isolate traffic sessions by users or by queues

* remove extra indices

* corrections

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2023-01-11 17:47:20 +04:00
Evgeny Poberezkin
8d9816809f simplify queue rotation protocol (#550)
* simplify queue rotation protocol

* use simplified rotation protocol, update tests

* simplify schema

* delete all connection queues

* refactor

* switch notifications to the new queue, tests

* remove TODO

* refactor

* rfc correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* remove duplicate set active

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-10-31 13:33:28 +04:00
Evgeny Poberezkin
eb5c1c78cb connection queue redundancy and rotation (#521)
* rfc: queue rotation

* update rfc

* messages for queue rotation

* allow multiple subscribed queues per connection in Agent/Client.hs

* refactor

* fix module name

* allow multiple queues in duplex connection type

* update commands

* add indices

* addConnectionRcvQueue

* switch connection to another queue (WIP)

* update schema/protocol

* switching queue works, but sending messages after the switch fails

* messages are delivered after rotation

* use connection-scoped queue ID

* rename queue records fields

* refactor using SMPQueue class/instances

* simplify queries

* QKEY: check queue is not secured, refactor

* update rfc

* mark queue as primary in QUSE

* queue rotation errors

* fix async ack

* fix async ACK to send OK

* correction

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* use SWCH command

* rename

* take into account only active queue subscription when determining connection result if at least one queue is active

* remove comment

* only enable notifications for connections with enableNtfs = True

* async test (WIP)

* async queue rotation test

* simplify combining results

* test with 2 servers

* fix unused subscribeConnection

* switch to cabal build

* increase build timeout

* increase delay in async test

* skip queue rotation tests

* build matrix

* step name

* use ubuntu-18.04 in build matrix

* enable rotation tests

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-10-29 18:57:01 +01:00
Evgeny Poberezkin
e4b77ed9e6 use SQLCipher (#507)
* use SQLCipher

* pass database key via options, use local direct-sqlcipher and sqlcipher-simple

* update stack.yaml

* use dependencies in git

* update sqlcipher dependencies
2022-08-30 12:31:41 +01:00
Evgeny Poberezkin
212a9bc4ab enable/disable connection notifications (#501)
* enable/disable connection notifications

* filter connections when intializing/disabling subscriptions

* fix

* fix 2

* add delay

* increase test delay

* increase test delay

* fix

* increase test delays

* increase test delays
2022-08-19 15:23:57 +01:00
Evgeny Poberezkin
68138c08d2 support multiple server hostnames per queue (#494)
* store SMP client version per queue

* use versioned format for queue addresses (not compatible with v1)

* multiple server hosts in queue URI/data, versioned encoding (cross-version tests fail)

* choose server host based on socksProxy setting

* test works

* multiple server addresses mostly work

* add onion hosts for provided servers

* update SMP hosts
2022-08-12 11:32:04 +01:00
JRoberts
7a72488275 ntf: use separate key to encrypt NMsgMeta (#410)
* ntf: use separate key to encrypt NMsgMeta

* key negotiation

* save key on server, use for encryption

* refactor?

* store error

* NtfQueueCreds

* server - NtfCreds

* comment, rename

* fix type

* ClientNtfCreds

* encoding
2022-06-20 17:16:26 +04:00
Evgeny Poberezkin
d1db7d6f79 withStore runs in a single transaction (#408)
* remove store typeclass

* make withStore run single transaction

* combine 2 store calls

* refactor

* add exports

* fix catch

* correction
2022-06-18 18:01:04 +01:00
Evgeny Poberezkin
5f94240ed4 remove DB connection pool to prepare for using semaphores (#400) 2022-06-15 14:11:03 +01:00
JRoberts
f7bd8a3982 ntf: agent ntf workers implementation, store methods (#394) 2022-06-14 14:34:18 +04:00
Evgeny Poberezkin
c1348aa54f SMP connection handshake v2 (#390)
* SMP connection handshake v2

* hadshake v2 [mostly] works

* all tests pass, some race conditions remain

* fix build

* fix race conditions, send CON after all HELLOs in duplexHandshake mode

* add comments

* comment

* add comments
2022-06-09 13:47:07 +01:00
Evgeny Poberezkin
4b3d04bd27 support message flags visible to SMP server to control notifications (and for any future extensions) (#386)
* support stopping and resuming agent  (#385)

* export agentDbPath

* support fully closing and resuming agent

* whitespace

* clean up

* support message flags visible to SMP server to control notifications (and for any future extensions)

* simplify message flags encoding

* GET command
2022-06-06 12:59:45 +01:00
Evgeny Poberezkin
7a19ab224b make sending confirmation asynchronous (#327)
* make sending confirmation asynchronous

* confirm first queue synchronously (on JOIN, and delete connection on failure), second queue asynchronously (from REPLY msg)

* test to retry JOIN when the first attempt fails

* process confirmation delivery errors
2022-03-02 15:52:45 +00:00
Efim Poberezkin
c9994c3a2c add "yes to migrations" option (#316) 2022-02-07 11:51:39 +04:00
Evgeny Poberezkin
083d39be22 store missing port as empty string instead of NULL (#280) 2022-01-11 16:01:09 +00:00
Evgeny Poberezkin
0775bceb5e remove unnecessary separation of UnliftIO instance for ExceptT (#275) 2022-01-10 19:06:04 +00:00
Evgeny Poberezkin
2a89394174 integrate double ratchet into agent (#268)
* separate skipped messages from ratchet

* return diff for skipped messages instead of the whole state (tests fail)

* fix ratchet tests

* JSON encoding/decoding for ratchet state

* schema for ratchets

* split MonadUnliftIO instance for ExceptT to a separate file

* update StrEncoding instance for Str

* ratchet store methods

* updateRatchet store method

* move E2E ratchet params to Ratchet module

* x3dh key agreement for double ratchet

* test/fix x3dh, use x3dh for ratchets initialization

* store/get x3dh keys, save ratchet of fixed X448 type

* double-ratchet encryption integration (tests fail)

* fix double ratchet

* fix padding and message length

* remove unused code for "activations"

* remove comment

* add version checks for forward/backward compatibility

* split loading ratchet and skipped message keys

* remove unused encoding instances for Algorithm types

* update ratchet initialization params
2022-01-10 12:01:54 +00:00
Efim Poberezkin
5e57c5dfd7 rename remaining occurrences of connection alias into connection id (#265) 2022-01-07 15:37:40 +04:00
Efim Poberezkin
d87ff42cf5 start with internal message id 1; fix id ordering in tests (#264) 2022-01-07 14:52:37 +04:00
Evgeny Poberezkin
488398df9f change message envelopes and encoding, unify message delivery (#252)
* types and encodings for double ratchet integration

* upgrade stack resolver

* type classes for version agreement, encode/decode connection request links and E2E params with versioning

* encode/decode client parameters (version and DH key) in SMP queue URI using query string parameters

* restore support of the current SMP queue URI format

* update AMessage to only send queues in REPLY message (not the full connection request)

* new agent message evnvelopes (tests fail)

* new message envelopes - tests pass

* store fully encrypted messages before sending

* unify message delivery via DB queue (excluding confirmation and invitation)

* remove activateSecuredQueue

* linter hints

* remove comment

* export order

* save rachet-encrypted message, not per-queue encrypted

* delete message after it is accepted by the server, reduce message delivery interval for the tests

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>
2022-01-05 19:52:37 +00:00
Evgeny Poberezkin
4a73a7ecd4 remove public header from all messages, fix envelope sizes (#251)
* fix envelope sizes

* only send sender DH pub key for per-queue E2E with confirmation message
2022-01-03 14:33:30 +00:00
Evgeny Poberezkin
c66f9efa55 merge migrations, prepare schema for multiple queues per connection (#243)
* merge migrations into initial

* update connection/queue tables in schema, simplify creating connections

* refactor create connection

* remove queue fields from connections table

* update server constraints
2021-12-29 17:55:27 +00:00
Evgeny Poberezkin
73cad5a6c4 simple per-queue e2e encryption with NaCl crypto_box (#242)
* simple per-queue e2e encryption with NaCl crypto_box

* add e2e keys and DH secrets to schema

* agree and save shared DH secret per queue (not used yet)

* protocol changes for uniform padding and message part lengths

* correct message structure diagrams

* make per-queue E2E encryption non-optional

* refactor crypto keys

* use NaCl crypto_box for per-queue E2E encryption, remove RSA keys from queues

* remove RSA support

* merge migration with E2E DH keys

* clean up

* remove unused methods

* parsing/serializing agent messages

* remove sender timestamp from DB and code

* clean up

* slean up

* s/SMPConfMsg/SMPConfirmation/

* serializeAgentMessage = serializeClientMessage . agentToClientMsg

* simplify error handling

* update protocol docs
2021-12-29 14:27:10 +00:00
Evgeny Poberezkin
323fb1f03c remove SMP server signing responses (#226)
* remove SMP server signing responses

* keep only one session ID
2021-12-16 07:15:45 +00:00
Evgeny Poberezkin
bfa05c9432 all tests pass! 2021-12-12 21:17:25 +00:00
Evgeny Poberezkin
f640e71f82 remove BlockArguments extension (#220) 2021-12-10 12:21:54 +00:00
Evgeny Poberezkin
b116408ec3 Merge branch 'master' into v5 2021-12-04 18:55:23 +00:00
Evgeny Poberezkin
04c65d98da support permanent connection link ("contact") in SMP agent protocols (#143)
* open/public queue extension for SMP and SMP agent protocols

* add connection mode - invitation or contact

* use ConnectionMode with REQ and ACPT agent notification/command

* parameterize ConnectionRequest with ConnectionMode

* implement Contact connection mode for permanent connection links

* tests for contact connections
2021-12-04 18:42:08 +00:00
Evgeny Poberezkin
e1002d5ac0 GADTs for cryptographic keys (#208)
* GADTs for cryptographic keys

* update tests (signature timing tests still fail)

* fix signature verification timing tests

* configurable algorithm to sign commands to SMP queues (Ed448 by default)

* add dummy Ed keys, add timing tests for Ed keys

* re-enable Connection subscriptions tests
2021-11-28 11:44:22 +00:00
Evgeny Poberezkin
99b3749890 simplify RSA private key types (#207)
* simplify RSA private key types

* remove updateSignKey method
2021-11-28 07:08:47 +00:00
Evgeny Poberezkin
dd5137c336 asynchronous message sending to allow accepting messages when agent (or server) is offline (#179)
* asynchronous message sending to allow accepting messages when agent (or server) is offline

* send pending messages after agent restart; test for pending messages delivery when server restarted

* test pending message delivery after agent restart

* message delivery process per server

* test layout
2021-08-12 22:15:55 +01:00
Evgeny Poberezkin
3829479a21 Remove introductions, rename CONF/LET to REQ/ACPT (#175)
* remove intros and functions in Reader monad

* rename CONF/LET to REQ/ACPT, allowConnection to acceptConnection
2021-08-05 08:27:44 +01:00
Efim Poberezkin
3d9ceff691 ask client for confirmation of sender; make establishment of connection asynchronous (#163)
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2021-07-04 04:48:24 +10:00
Evgeny Poberezkin
daad3315eb upgrade stack resolver to lts-18.0 (#165)
* upgrade stack resolver to lts-18.0

* fix random package version
2021-06-30 10:35:50 +01:00
Evgeny Poberezkin
e4d9b481ec refactor message meta to MsgMeta type (#164) 2021-06-30 10:29:45 +01:00
Evgeny Poberezkin
7af7272635 SQLiteStore connection pool (#162)
* SQLiteStore connection pool

* move SQLiteStore to agent env - single store instance is used for all clients
2021-06-25 18:17:11 +01:00
Evgeny Poberezkin
09c6adeabc pass migration as parameter (to use in simplex-chat) (#161)
* pass migration as parameter (to use in simplex-chat)

* add connId parameter to create/joinConnection
2021-06-24 18:39:59 +01:00
Evgeny Poberezkin
ab89963f45 introduction protocol (#156)
* commands to support introduction

* agent messages / envelopes to support introductions

* introductions and invitations table; insert record with random unique ID

* store class methods and types for introductions

* process INTRO and ACPT commands for connection introductions

* fix tests: add MonadFail constraint, remove OK response to JOIN

* process agent messages for introductions

* ICON notification when introduction is completed

* replace multiway if with case

* correction

* support random connection IDs

* save additional connection fields, refactor create connection funcs

* refactor

* refactor

* test duplex connection with random IDs

* store methods for introductions

* test introduction

* fix parsing of CON agent message

* test introduction with random connection IDs

* broadcast with random connection and broadcast IDs

* clean up sql
2021-06-11 21:33:13 +01:00
Efim Poberezkin
8bd42035aa reorganize sqlite store test suite (#138) 2021-05-10 15:10:32 +04:00
Evgeny Poberezkin
5d59e4b2bd package and module docs, remove Simplex.Markdown (moved to simplex-chat), rename Agent.Transmission to Agent.Protocol (#133)
* package and module docs, remove Simplex.Markdown (moved to simplex-chat), rename Agent.Transmission to Agent.Protocol

* move errors.md to haddock comments, Transport docs

* add CHANGELOG.md, add missing package versions

* changelog, copyright

* docs for Simplex.Messaging.Crypto

* consistent punctuation

* use absolute URLs in readme

* correction
2021-05-09 09:36:08 +01:00
Efim Poberezkin
ad87442811 sqlite busy stress test (#131)
* sqlite busy stress test

* refactor withStore2

* refactor withStore2

* pragmas

* swith to IMMEDIATE sqlite transactions and add retry on ErrorBusy

* refactor

* increase timeout, print errors and results

* remove logging errors/results

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2021-05-08 16:30:40 +01:00