92 Commits

Author SHA1 Message Date
Evgeny Poberezkin c74f4d729b create/verify agent schema during tests (#374)
* create/verify agent schema during tests

* add --indent to .schema, enable all tests

* remove -threaded from test
2022-05-24 19:28:40 +01:00
Evgeny Poberezkin 1064e9c315 disconnect inactive clients (#366)
* disconnect inactive clients

* update test description

* use one thread per client to disconnect inactive clients

* remove comments

* more comments
2022-04-30 10:56:01 +01:00
Evgeny Poberezkin c6dde772b4 batch all connections in DOWN/UP agent messages (#363) 2022-04-25 08:26:52 +01:00
Evgeny Poberezkin e6fbaf5e50 primary tokens (#356)
* primary tokens

* support repeat token registration (TODO fix tests)

* fix notifications tests

* fix/test repeat/new registrations of the same token

* re-register token when subsequent ntf command fails with AUTH error (e.g. when server is re-started)

* cancel periodic notifications when token is deleted on the server

* debug failing test on CI

* fix notification test in CI

* debug CI test

* add delay in notificaitons test after server restart
2022-04-21 17:04:26 +01:00
Evgeny Poberezkin 17888f89a9 test notification token with agent and notifications server (#353)
* test notification token with agent and notifications server

* notification server test with APNS mock

* set environment variables in the test

* use base64url encoding in encrypted notification data
2022-04-18 08:57:55 +01:00
Evgeny Poberezkin 9d8a9c4fe4 interval notifications (TCRN command) (#352)
* notifications: periodic notifications

* agent: allow repeat token registrations, delete old tokens from notification server (e.g., when database is moved to another device)

* decrypt token verification code in the agent

* check token status, send TCRN on registration if it was enabled

* fix http2/apns response handling for error responses (also, APNS seems not to send content-length header?)
2022-04-17 10:34:51 +01:00
Evgeny Poberezkin f577fcdacf agent schema/methods/types/store methods for notifications tokens (#348)
* agent schema/methods/types/store methods for notifications tokens

* register notification token on the server

* agent commands for notification tokens

* refactor initial servers from AgentConfig

* agent store functions for notification tokens

* server STM store methods for tokens

* fix protocol client for ntfs (use generic handshake), minimal server and agent tests

* server command to verify ntf token
2022-04-08 08:47:04 +01:00
Evgeny Poberezkin d31958855f ntf server implementation, updated ntf protocol, ntf client based on refactored protocol client, bare-bones SMP agent to manage ntf connections (to connect to ntf server) (#338)
* process ntf server commands

* when subscription is re-created and it was ENDed, resubscribe to SMP

* SMPClientAgent draft

* SMPClientAgent: remove double tracking of subscriptions

* subscriber frame

* PING error now throws error to restart SMPClient for more reliable re-connection (#342)

* increase TCP timeout to 5 sec

* add pragmas and vacuum db (#343)

* vacuum in each connection to enable auto-vacuum (#344)

* update protocol, token verification

* refactor SMPClient to ProtocoClient, to use with notification server protocol

* notification server client, managing notification clients in the agent

* stub for push payload

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-04-02 16:14:19 +01:00
Evgeny Poberezkin 7a19ab224b make sending confirmation asynchronous (#327)
* make sending confirmation asynchronous

* confirm first queue synchronously (on JOIN, and delete connection on failure), second queue asynchronously (from REPLY msg)

* test to retry JOIN when the first attempt fails

* process confirmation delivery errors
2022-03-02 15:52:45 +00:00
Efim Poberezkin c9994c3a2c add "yes to migrations" option (#316) 2022-02-07 11:51:39 +04:00
Efim Poberezkin 6e7089284e cease attempts to send HELLO after timeout (#298) 2022-01-18 13:50:43 +04:00
Evgeny Poberezkin 083d39be22 store missing port as empty string instead of NULL (#280) 2022-01-11 16:01:09 +00:00
Evgeny Poberezkin 0775bceb5e remove unnecessary separation of UnliftIO instance for ExceptT (#275) 2022-01-10 19:06:04 +00:00
Evgeny Poberezkin 2a89394174 integrate double ratchet into agent (#268)
* separate skipped messages from ratchet

* return diff for skipped messages instead of the whole state (tests fail)

* fix ratchet tests

* JSON encoding/decoding for ratchet state

* schema for ratchets

* split MonadUnliftIO instance for ExceptT to a separate file

* update StrEncoding instance for Str

* ratchet store methods

* updateRatchet store method

* move E2E ratchet params to Ratchet module

* x3dh key agreement for double ratchet

* test/fix x3dh, use x3dh for ratchets initialization

* store/get x3dh keys, save ratchet of fixed X448 type

* double-ratchet encryption integration (tests fail)

* fix double ratchet

* fix padding and message length

* remove unused code for "activations"

* remove comment

* add version checks for forward/backward compatibility

* split loading ratchet and skipped message keys

* remove unused encoding instances for Algorithm types

* update ratchet initialization params
2022-01-10 12:01:54 +00:00
Efim Poberezkin 5e57c5dfd7 rename remaining occurrences of connection alias into connection id (#265) 2022-01-07 15:37:40 +04:00
Efim Poberezkin d87ff42cf5 start with internal message id 1; fix id ordering in tests (#264) 2022-01-07 14:52:37 +04:00
Evgeny Poberezkin 488398df9f change message envelopes and encoding, unify message delivery (#252)
* types and encodings for double ratchet integration

* upgrade stack resolver

* type classes for version agreement, encode/decode connection request links and E2E params with versioning

* encode/decode client parameters (version and DH key) in SMP queue URI using query string parameters

* restore support of the current SMP queue URI format

* update AMessage to only send queues in REPLY message (not the full connection request)

* new agent message evnvelopes (tests fail)

* new message envelopes - tests pass

* store fully encrypted messages before sending

* unify message delivery via DB queue (excluding confirmation and invitation)

* remove activateSecuredQueue

* linter hints

* remove comment

* export order

* save rachet-encrypted message, not per-queue encrypted

* delete message after it is accepted by the server, reduce message delivery interval for the tests

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>
2022-01-05 19:52:37 +00:00
Evgeny Poberezkin 4a73a7ecd4 remove public header from all messages, fix envelope sizes (#251)
* fix envelope sizes

* only send sender DH pub key for per-queue E2E with confirmation message
2022-01-03 14:33:30 +00:00
Evgeny Poberezkin f3523bbba9 make KeyHash non-optional, verify KeyHash in SMP handshake, use StrEncoding class (#250)
* make KeyHash non-optional, StrEncoding class

* change server URI format in agent config, refactor with StrEncoding

* refactor Crypto using checkAlgorithm

* refactor parsing connection requests

* prepare to validate CA fingerprint sent in client handshake

* KeyHash check in handshake

* rename type to CliCommand

* server validates keyhash sent by the client

* validate -a option when parsing

* more of StrEncoding
2022-01-02 22:24:43 +00:00
Evgeny Poberezkin 5e29e3698e binary SMP protocol encoding, split Command type to two types (#245)
* binary SMP protocol encoding (server tests fail)

* use 1 byte for bytestring length when encoding/decoding

* Encoding class, binary tags

* update server tests

* negotiate SMP version in client/server handshake

* add version columns to queues and connections

* split parsing SMP client commands and server responses to different functions

* check uniqueness of protocol tags

* split client commands and server responses/messages to separate types

* update types in SMP client

* remove pattern synonyms for SMP errors

* simplify getHandshake

* update SMP protocol encoding in protocol spec

* encode time as a number of seconds (64-bit integer) since epoch
2022-01-01 13:10:19 +00:00
Evgeny Poberezkin c66f9efa55 merge migrations, prepare schema for multiple queues per connection (#243)
* merge migrations into initial

* update connection/queue tables in schema, simplify creating connections

* refactor create connection

* remove queue fields from connections table

* update server constraints
2021-12-29 17:55:27 +00:00
Evgeny Poberezkin 73cad5a6c4 simple per-queue e2e encryption with NaCl crypto_box (#242)
* simple per-queue e2e encryption with NaCl crypto_box

* add e2e keys and DH secrets to schema

* agree and save shared DH secret per queue (not used yet)

* protocol changes for uniform padding and message part lengths

* correct message structure diagrams

* make per-queue E2E encryption non-optional

* refactor crypto keys

* use NaCl crypto_box for per-queue E2E encryption, remove RSA keys from queues

* remove RSA support

* merge migration with E2E DH keys

* clean up

* remove unused methods

* parsing/serializing agent messages

* remove sender timestamp from DB and code

* clean up

* slean up

* s/SMPConfMsg/SMPConfirmation/

* serializeAgentMessage = serializeClientMessage . agentToClientMsg

* simplify error handling

* update protocol docs
2021-12-29 14:27:10 +00:00
Evgeny Poberezkin 51a9750891 double ratchet algorithm implementation (#236)
* started double ratchet implementation

* initialize ratchets

* started ratchet encryption

* ratchet encryption

* simplify / narrow down Ratchet type

* double ratchet decryption "framework"

* advance receive ratched on skipped messages

* more ratchet decryption

* double ratchet encrypt/decrypt (TODO tests)

* double ratchet tests

* double ratchet tests

* use ratchet AD in header encryption, use header and ratchet AD as AD in message encryption

* change ratchet message error, remove Show instances

* Update tests/AgentTests/DoubleRatchetTests.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* Update tests/AgentTests/DoubleRatchetTests.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* Update tests/AgentTests/DoubleRatchetTests.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* Update tests/AgentTests/DoubleRatchetTests.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* Update tests/AgentTests/DoubleRatchetTests.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* Update src/Simplex/Messaging/Crypto/Ratchet.hs

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>

* test in the same ratchet step

* merge tests

* Update src/Simplex/Messaging/Crypto/Ratchet.hs

* Update src/Simplex/Messaging/Crypto/Ratchet.hs

* remove HMAC comment

Co-authored-by: Efim Poberezkin <8711996+efim-poberezkin@users.noreply.github.com>
2021-12-25 17:13:53 +00:00
Efim Poberezkin e2cd370513 certificate validation on client side; check stored fingerprint on server start-up; non-optional fingerprint parsing (#234, closes #155)
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2021-12-22 23:24:58 +04:00
Evgeny Poberezkin 323fb1f03c remove SMP server signing responses (#226)
* remove SMP server signing responses

* keep only one session ID
2021-12-16 07:15:45 +00:00
Evgeny Poberezkin bfa05c9432 all tests pass! 2021-12-12 21:17:25 +00:00
Evgeny Poberezkin f640e71f82 remove BlockArguments extension (#220) 2021-12-10 12:21:54 +00:00
Evgeny Poberezkin 58758e0572 Merge branch 'master' into v5 2021-12-04 21:08:38 +00:00
Evgeny Poberezkin 3e226fc3f2 add CONF/LET for invitations, use REQ/ACPT only with contact connections (#212) 2021-12-04 21:08:02 +00:00
Evgeny Poberezkin b116408ec3 Merge branch 'master' into v5 2021-12-04 18:55:23 +00:00
Evgeny Poberezkin 04c65d98da support permanent connection link ("contact") in SMP agent protocols (#143)
* open/public queue extension for SMP and SMP agent protocols

* add connection mode - invitation or contact

* use ConnectionMode with REQ and ACPT agent notification/command

* parameterize ConnectionRequest with ConnectionMode

* implement Contact connection mode for permanent connection links

* tests for contact connections
2021-12-04 18:42:08 +00:00
Evgeny Poberezkin 6126e72d4f Merge branch 'master' into v5 2021-12-02 18:06:42 +00:00
Evgeny Poberezkin 606bbf5062 URI syntax for SMP queues and connection requests (#210)
* URI syntax for SMP queues and connection requests

* fix connection request parsing/serializing

* add ConnectionRequest sqlite instances
2021-12-02 17:43:05 +00:00
Evgeny Poberezkin e1002d5ac0 GADTs for cryptographic keys (#208)
* GADTs for cryptographic keys

* update tests (signature timing tests still fail)

* fix signature verification timing tests

* configurable algorithm to sign commands to SMP queues (Ed448 by default)

* add dummy Ed keys, add timing tests for Ed keys

* re-enable Connection subscriptions tests
2021-11-28 11:44:22 +00:00
Evgeny Poberezkin 99b3749890 simplify RSA private key types (#207)
* simplify RSA private key types

* remove updateSignKey method
2021-11-28 07:08:47 +00:00
Evgeny Poberezkin 9a3afda290 ACK command, ackMessage, do not send ACK to SMP server on MSG (after the initial handshake) (#187) 2021-09-01 08:29:12 +01:00
Evgeny Poberezkin 7b5ebf7bd2 change type of agent sendMessage result to AgentMsgId (#183) 2021-08-26 23:25:49 +01:00
Evgeny Poberezkin dd5137c336 asynchronous message sending to allow accepting messages when agent (or server) is offline (#179)
* asynchronous message sending to allow accepting messages when agent (or server) is offline

* send pending messages after agent restart; test for pending messages delivery when server restarted

* test pending message delivery after agent restart

* message delivery process per server

* test layout
2021-08-12 22:15:55 +01:00
Evgeny Poberezkin 3829479a21 Remove introductions, rename CONF/LET to REQ/ACPT (#175)
* remove intros and functions in Reader monad

* rename CONF/LET to REQ/ACPT, allowConnection to acceptConnection
2021-08-05 08:27:44 +01:00
Efim Poberezkin 5e380e1d47 add tests for establishing connection asynchronously; retrieve correct key for activation on restart (#169) 2021-07-10 20:02:48 +10:00
Efim Poberezkin 3d9ceff691 ask client for confirmation of sender; make establishment of connection asynchronous (#163)
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2021-07-04 04:48:24 +10:00
Evgeny Poberezkin daad3315eb upgrade stack resolver to lts-18.0 (#165)
* upgrade stack resolver to lts-18.0

* fix random package version
2021-06-30 10:35:50 +01:00
Evgeny Poberezkin e4d9b481ec refactor message meta to MsgMeta type (#164) 2021-06-30 10:29:45 +01:00
Evgeny Poberezkin 7af7272635 SQLiteStore connection pool (#162)
* SQLiteStore connection pool

* move SQLiteStore to agent env - single store instance is used for all clients
2021-06-25 18:17:11 +01:00
Evgeny Poberezkin 09c6adeabc pass migration as parameter (to use in simplex-chat) (#161)
* pass migration as parameter (to use in simplex-chat)

* add connId parameter to create/joinConnection
2021-06-24 18:39:59 +01:00
Evgeny Poberezkin ab89963f45 introduction protocol (#156)
* commands to support introduction

* agent messages / envelopes to support introductions

* introductions and invitations table; insert record with random unique ID

* store class methods and types for introductions

* process INTRO and ACPT commands for connection introductions

* fix tests: add MonadFail constraint, remove OK response to JOIN

* process agent messages for introductions

* ICON notification when introduction is completed

* replace multiway if with case

* correction

* support random connection IDs

* save additional connection fields, refactor create connection funcs

* refactor

* refactor

* test duplex connection with random IDs

* store methods for introductions

* test introduction

* fix parsing of CON agent message

* test introduction with random connection IDs

* broadcast with random connection and broadcast IDs

* clean up sql
2021-06-11 21:33:13 +01:00
Efim Poberezkin 8bd42035aa reorganize sqlite store test suite (#138) 2021-05-10 15:10:32 +04:00
Evgeny Poberezkin 5d59e4b2bd package and module docs, remove Simplex.Markdown (moved to simplex-chat), rename Agent.Transmission to Agent.Protocol (#133)
* package and module docs, remove Simplex.Markdown (moved to simplex-chat), rename Agent.Transmission to Agent.Protocol

* move errors.md to haddock comments, Transport docs

* add CHANGELOG.md, add missing package versions

* changelog, copyright

* docs for Simplex.Messaging.Crypto

* consistent punctuation

* use absolute URLs in readme

* correction
2021-05-09 09:36:08 +01:00
Efim Poberezkin ad87442811 sqlite busy stress test (#131)
* sqlite busy stress test

* refactor withStore2

* refactor withStore2

* pragmas

* swith to IMMEDIATE sqlite transactions and add retry on ErrorBusy

* refactor

* increase timeout, print errors and results

* remove logging errors/results

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2021-05-08 16:30:40 +01:00
Evgeny Poberezkin 7aacee405e agent: send CON to user when the 1st party responds HELLO; fix REPLY vulnerability (#130)
* agent: send CON to user when the 1st party responds HELLO; fix REPLY vulnerability

* test for getRcvConn

* add commented OK response to JOIN command

* store: use Only newtype to select one field
2021-05-07 06:57:53 +01:00