Compare commits

...
722 Commits
Author SHA1 Message Date
sh 9a0a85a24c feat(bench): self-contained Docker Compose setup
Build benchmark binary inside container via multi-stage
Dockerfile. All-in-one: docker compose run bench.
2026-03-20 14:51:16 +00:00
sh 2a0af04ab8 feat(bench): add smp-server memory benchmark framework
Layered benchmark that isolates per-component memory cost:
- Phase 1: baseline (no clients)
- Phase 2: TLS connections only
- Phase 3: queue creation (NEW + KEY)
- Phase 4: subscriptions (SUB)
- Phase 5: message send
- Phase 6: message receive + ACK
- Phase 7: sustained load with time-series

Includes Docker Compose (PostgreSQL 17), run.sh with
--compare-rts mode for testing different GC configurations.
2026-03-20 14:48:11 +00:00
sh 4c8ace4db6 update findings 2026-03-20 12:01:01 +00:00
sh 323be9c6a4 feat(smp-server): add per-client and RTS memory metrics
Add clientSubs, clientSndQ, clientMsgQ, clientThreads counts
and GHC RTS large objects, compact, fragmentation metrics
to identify memory growth not explained by server-level maps.
2026-03-20 12:01:01 +00:00
sh c6d6e30e48 add memory-analysis-results based on the produced logs 2026-03-20 12:01:01 +00:00
sh 646476f5fa smp-server: add periodic memory diagnostics logging
Log sizes of all in-memory data structures every 5 minutes
to help identify memory growth root cause on busy servers.
2026-03-20 12:01:01 +00:00
sh 404dd10d4a further analysis 2026-03-20 12:01:01 +00:00
sh 931b1cc725 smp-server: memory usage analysis 2026-03-20 12:01:01 +00:00
Evgeny Poberezkin 1a255f2e2f Merge branch 'master' into rcv-services 2026-03-20 09:00:01 +00:00
Evgeny 963d7b2f75 fix small bugs (#1740)
* fix small bugs

* toChunks
2026-03-20 08:59:38 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 8f4274763b smp: service fixes (#1737)
* smp: deliver service subscription to correct client

* tests: more resilient to concurrency

* optimize PostgreSQL query

* fix service re-association after server "downgrade"

* correctly handle service removed from server (and ID changed)

* remove unused

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-20 07:54:26 +00:00
Evgeny Poberezkin 4e4e0a4f42 6.5.0.11 2026-03-17 09:26:02 +00:00
sh 082a6c6f22 web: serve on-the-fly compressed gzip static files (#1735)
* web: serve pre-compressed gzip static files

* web: compress static files on the fly instead of pre-compressed
2026-03-16 09:08:43 +00:00
shandEvgeny dc2921e4ce xftp-server: embed file download widget in XFTP server web page (#1733)
* xftp-server: embed file download widget in XFTP server web page

When a URL has a hash fragment (>50 chars), the server page shows the
file download UI instead of the server info page. Embeds xftp-web
assets (JS, CSS, crypto worker) and protocol overlay with matching
website content. Overlay renders below the server navbar.

* xftp-server: fix overlay scroll lock, remove extra margin, fix dark SVG

* xftp-server: move file transfer widget to standalone /file page

* web: collapse all repeated Nothing sections in render

section_ only collapsed the first occurrence of a section when content
was Nothing, leaving subsequent sections with the same label intact.
This caused SMP server pages to show raw <x-xftpConfig> tags.

* xftp-server: update bundled css/js

* xftp-server: move file.html to xftp-server, rename xftp bundle dir

* web: remove unused server-info wrapper div

* refactor

* fix

---------

Co-authored-by: Evgeny <evgeny@poberezkin.com>
2026-03-13 16:00:02 +00:00
sh 5a32e729e0 xftp-web: add "Upload your file" link after download completes (#1736)
* xftp-web: add "Upload your file" link after download completes

* xftp-web: call initUpload directly instead of hashchange dispatch
2026-03-13 15:01:15 +00:00
Evgeny Poberezkin 48eba591bc Merge branch 'master' into rcv-services 2026-03-12 17:05:51 +00:00
shandEvgeny Poberezkin 782cacfb3c fix: using simplexmq as dependency (move embedFile to executables) (#1734)
* web: parameterize generateSite, remove Embedded from library

Move embedFile/embedDir out of the library so it works when
simplexmq is consumed as a dependency. generateSite now accepts
mediaContent, wellKnown, and linkHtml as parameters.

* smp-server, xftp-server: embed static files in executables

Add shared apps/common/Embedded.hs with TH splices, update SMPWeb
and XFTPWeb to pass embedded content to generateSite, move
file-embed dependency from library to executables and test suite.

* refactor

* add export, move common files to Web subfolder

* fix .cabal

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-12 17:05:00 +00:00
sh 328d3b941a xftp-web: use XFTP server domain in share link, verify on download (#1732)
* xftp-web: use XFTP server domain in share link, verify on download

Use a random XFTP server host from the file description as the link
origin instead of the current page domain. On download, verify
that the hosting domain matches one of the servers in the description.

* xftp-web: use first description server as link origin

* xftp-web: show wrong server error in download UI instead of blank page
2026-03-11 15:59:26 +00:00
sh b6b87a6323 xftp-web: disable minification in vite build (#1731) 2026-03-11 09:08:23 +00:00
Evgeny @ SimpleX Chat b81670c414 docs: move implemented rfcs 2026-03-10 19:18:34 +00:00
Evgeny @ SimpleX Chat 98351cf097 docs: add missing encodings and other protocol corrections 2026-03-10 16:36:24 +00:00
Evgeny @ SimpleX Chat 01785d5de8 docs: add e2e encrypted message wire encoding to PQDR spec 2026-03-10 09:04:21 +00:00
Evgeny @ SimpleX Chat f745ce5ab6 docs: fix minor issues in protocols 2026-03-10 08:15:53 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 583f4e059d update RFCs (#1730)
* update RFCs

* update

* update overview

* update terminology

* original language in threat model

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-09 23:35:41 +00:00
Evgeny @ SimpleX Chat 3c5752383d update protocol docs 2026-03-09 16:29:05 +00:00
sh 33454444a4 xftp-web: new version (#1728)
* xftp-web: remove flux from preset

* xftp-web: new version
2026-03-09 14:26:40 +00:00
Evgeny Poberezkin 3a25561c78 Merge branch 'master' into rcv-services 2026-03-09 12:25:52 +00:00
Evgeny Poberezkin 1449e6418c 6.5.0.10 2026-03-09 12:25:08 +00:00
sh 3a627a9599 tests: use correct web CA certificate in CLI static files test (#1727) 2026-03-09 12:23:03 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> d2d834ad16 agent: validate destination relay certificate, allow 3-4 certificate chains (#1717)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-09 12:22:42 +00:00
Evgeny @ SimpleX Chat c624a10e70 Merge branch 'master' into rcv-services 2026-03-09 10:23:01 +00:00
shandEvgeny Poberezkin 437cdde4a5 xftp: add web page for server information (#1724)
* xftp: add web page for server information

* web: rename XFTP.Web to XFTPWeb, remove XFTP subdirectory

* refactor(xftp): remove storage quota from web page

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-09 08:44:28 +00:00
shandEvgeny Poberezkin eed1bf14c6 web: extract shared web module from smp-server (#1723)
* web: extract shared web module from smp-server

Move web serving infrastructure (warp, static files, HTML templating)
from apps/smp-server/web/Static.hs into library modules:
- Simplex.Messaging.Server.Web (generic web infra + templating)
- Simplex.Messaging.Server.Web.Embedded (TH-embedded assets)

Move static assets from apps/smp-server/static/ to
src/Simplex/Messaging/Server/Web/.

Move EmbeddedWebParams/WebHttpsParams from Server.Main to Server.Web.

Keep SMP-specific rendering (serverInformation) in apps/smp-server/SMP/Web.hs.

generateSite is now generic: takes pre-rendered HTML + link page paths,
enabling reuse by XFTP and NTF servers.

* web: add tests for templating engine

Tests for render, section_, item_, and timedTTLText functions
in Simplex.Messaging.Server.Web module.

* web: add serverInfoSubsts, serveStaticPageH2, safe port parsing

* web: rename SMP.Web to SMPWeb, remove SMP subdirectory

* fix(web): section_ collapsing sections with Just "" content

Commit e48bedea ("servers: fix server pages when source code is not
specified") changed section_ to treat Just "" the same as Nothing -
collapsing the section. The intent was to handle the sourceCode case
(empty string when not specified), but the guard
`not (B.null content)` also broke operator, admin, complaints, and
hosting - all of which legitimately use Just "" as a
section-present marker.

Before (correct):
  Nothing -> before <> next
  Just content -> before <> item_ label content inside <> ...

After (broken):
  Just content | not (B.null content) -> ...
  _ -> before <> next

Restore the original behavior: only Nothing collapses a section.

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-09 08:42:38 +00:00
Evgeny @ SimpleX Chat 313e96513c docs: correction to governance process (#1725) 2026-03-08 17:27:56 +00:00
Evgeny @ SimpleX Chat 8518f6087b docs: agent threat model 2026-03-07 15:47:46 +00:00
Evgeny Poberezkin 1d3057985c Merge branch 'master' into rcv-services 2026-03-04 07:33:20 +00:00
Evgeny c6e3a4d80f add missing exports (#1722)
* add missing exports

* fix dependency
2026-03-04 07:31:46 +00:00
Evgeny Poberezkin aebc01ba58 Merge branch 'master' into rcv-services 2026-03-03 21:16:46 +00:00
Evgeny Poberezkin 0e67647e90 6.5.0.9 2026-03-02 18:14:37 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> f3408d9bb6 explicit exports (#1719)
* explicit exports

* more empty exports

* add exports

* reorder

* use correct ControlProtocol type for xftp router

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-02 17:34:01 +00:00
sh 3ecb901e3e xftp-web: fix build and Playwright test failures (#1720) 2026-03-02 16:22:36 +00:00
EvgenyEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>shumsh
f6aca47604 xftp: implementation of XFTP client as web page (#1708)
* xftp: implementation of XFTP client as web page (rfc, low level functions)

* protocol, file descriptions, more cryptogrpahy, handshake encoding, etc.

* xftp server changes to support web slients: SNI-based certificate choice, CORS headers, OPTIONS request

* web handshake

* test for xftp web handshake

* xftp-web client functions, fix transmission encoding

* support description "redirect" in agent.ts and cross-platform compatibility tests (Haskell <> TypeScript)

* rfc: web transport

* client transport abstraction

* browser environment

* persistent client sessions

* move rfcs

* web page plan

* improve plan

* webpage implementation (not tested)

* fix test

* fix test 2

* fix test 3

* fixes and page test plan

* allow sending xftp client hello after handshake - for web clients that dont know if established connection exists

* page tests pass

* concurrent and padded hellos in the server

* update TS client to pad hellos

* fix tests

* preview:local

* local preview over https

* fixed https in the test page

* web test cert fixtures

* debug logging in web page and server

* remove debug logging in server/browser, run preview xftp server via cabal run to ensure the latest code is used

* debug logging for page sessions

* add plan

* improve error handling, handle browser reconnections/re-handshake

* fix

* debugging

* opfs fallback

* delete test screenshot

* xftp CLI to support link

* fix encoding for XFTPServerHandshake

* support redirect file descriptions in xftp CLI receive

* refactor CLI redirect

* xftp-web: fixes and multi-server upload (#1714)

* fix: await sodium.ready in crypto/keys.ts (+ digest.ts StateAddress cast)

* multi-server parallel upload, remove pickRandomServer

* fix worker message race: wait for ready signal before posting messages

* suppress vite build warnings: emptyOutDir, externals, chunkSizeWarningLimit

* fix Haskell web tests: use agent+server API, wrap server in array, suppress debug logs

* remove dead APIs: un-export connectXFTP, delete closeXFTP

* fix TypeScript errors in check:web (#1716)

- client.ts: cast globalThis.process to any for browser tsconfig,
  suppress node:http2 import, use any for Buffer/chunks, cast fetch body
- crypto.worker.ts: cast sha512_init() return to StateAddress

* fix: serialize worker message processing to prevent OPFS handle race

async onmessage allows interleaved execution at await points.
When downloadFileRaw fetches chunks from multiple servers in parallel,
concurrent handleDecryptAndStore calls both see downloadWriteHandle
as null and race on createSyncAccessHandle for the same file,
causing intermittent NoModificationAllowedError.

Chain message handlers on a promise queue so each runs to completion
before the next starts.

* xftp-web: prepare for npm publishing (#1715)

* prepare package.json for npm publishing

Remove private flag, add description/license/repository/publishConfig,
rename postinstall to pretest, add prepublishOnly, set files and main.

* stable output filenames in production build

* fix repository url format, expand files array

* embeddable component: scoped CSS, dark mode, i18n, events, share

- worker output to assets/ for single-directory deployment
- scoped all CSS under #app, removed global resets
- dark mode via .dark ancestor class
- progress ring reads colors from CSS custom properties
- i18n via window.__XFTP_I18N__ with t() helper
- configurable mount element via data-xftp-app attribute
- optional hashchange listener (data-no-hashchange)
- completion events: xftp:upload-complete, xftp:download-complete
- enhanced file-too-large error mentioning SimpleX app
- native share button via navigator.share

* deferred init and runtime server configuration

- data-defer-init attribute skips auto-initialization
- window.__XFTP_SERVERS__ overrides baked-in server list

* use relative base path for relocatable build output

* xftp-web: retry resets to default state, use innerHTML for errors

* xftp-web: only enter download mode for valid XFTP URIs in hash

* xftp-web: render UI before WASM is ready

Move sodium.ready await after UI initialization so the upload/download
interface appears instantly. WASM is only needed when user triggers
an actual upload or download. Dispatch xftp:ready event once WASM loads.

* xftp-web: CLS placeholder HTML and embedder CSS selectors

Add placeholder HTML to index.html so the page renders a styled card
before JS executes, preventing layout shift. Use a <template> element
with an inline script to swap to the download placeholder when the URL
hash indicates a file download. Auto-compute CSP SHA-256 hashes for
inline scripts in the vite build plugin.

Change all CSS selectors from #app to :is(#app, [data-xftp-app]) so
styles apply when the widget is embedded with data-xftp-app attribute.

* xftp-web: progress ring overhaul

Rewrite progress ring with smooth lerp animation, green checkmark on
completion, theme reactivity via MutationObserver, and per-phase color
variables (encrypt/upload/download/decrypt).

Show honest per-phase progress: each phase animates 0-100% independently
with a ring color change between phases. Add decrypt progress callback
from the web worker so the decryption phase tracks real chunk processing
instead of showing an indeterminate spinner.

Snap immediately on phase reset (0) and completion (1) to avoid
lingering partial progress. Clean up animation and observers via
destroy() in finally blocks.

* xftp-web: single progress ring for upload, simplify ring color

* xftp-web: single progress ring for download

* feat(xftp-web): granular progress for encrypt/decrypt phases

Add byte-level progress callbacks to encryptFile, decryptChunks,
and sha512Streaming by processing data in 256KB segments. Worker
reports fine-grained progress across all phases (encrypt+hash+write
for upload, read+hash+decrypt for download). Progress ring gains
fillTo method for smooth ease-out animation during minimum display
delays. Encrypt/decrypt phases fill their weighted regions (0-15%
and 85-99%) with real callbacks, with fillTo covering remaining
time when work finishes under the 1s minimum for files >= 100KB.

* rename package

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: shum <github.shum@liber.li>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-03-02 09:57:46 +00:00
Evgeny Poberezkin 50ae1e1c3e docs: update governance process 2026-02-24 18:05:30 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 483ac674fb agent: fix possible deadlocks of queue overloading when processing messages (#1713)
* agent: fix possibly deadlocks of queue overloading when processing messages

* simplify

* refactor

* refactor

* refactor 2

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-02-24 16:47:00 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 3dba9c2b1e rfc: SimpleX Network Consortium governance (#1718)
* rfc: SimpleX Network Consortium governance

* update

* add draft label

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-02-22 18:58:09 +00:00
Evgeny Poberezkin 8fdc0703bc 6.5.0.8 2026-01-28 23:12:19 +00:00
Evgeny Poberezkin d7b90b8415 Merge branch 'stable' 2026-01-28 22:52:33 +00:00
Evgeny Poberezkin 9346b85c3f 6.4.8.0 2026-01-28 21:58:32 +00:00
Evgeny 3c5ec8d9a1 agent: improve error handling (#1707)
* agent: improve error handling

* simplify

* report critical error when subscriber crashes

* fix test
2026-01-28 21:54:41 +00:00
Evgeny 84e8b72ca3 docs: update protocol (#1705) 2026-01-27 21:21:54 +00:00
Evgeny d10e05b796 agent: split creating connection to two steps to prepare connection link in advance (#1704)
* agent: split creating connection to two steps to prepare connection link in advance

* linkEntityId, newOwnerAuth

* simplify
2026-01-27 10:54:13 +00:00
Evgeny 66cc06738e agent: fix tests for short links (#1703) 2026-01-26 00:11:50 +00:00
Evgeny Poberezkin 0f3b8a4a16 docs: update contributing 2026-01-24 18:05:01 +00:00
Evgeny Poberezkin ac825b0ff3 Merge branch 'master' into rcv-services 2026-01-24 14:10:51 +00:00
Evgeny a1596ed234 docs: contributing guide (#1702) 2026-01-24 13:33:17 +00:00
spaced4ndyandEvgeny Poberezkin 89b81d151f agent: getConnShortLinkAsync; implement joinConnectionAsync for contact connections; narrow setConnShortLinkAsync only to Contact connections (#1694)
* agent: getConnShortLinkAsync

* enable all tests

* comment

* comment

* join conn async for contact URI (wip)

* fix test

* remove enableNtfs param

* FixedLinkData

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-01-20 08:40:35 +00:00
Evgeny 3e5b654109 agent: support multiple link owners in link data (#1701)
* agent: support multiple link owners in link data

* fix
2026-01-19 22:08:11 +00:00
Evgeny Poberezkin 502d923817 agent: minor fixes 2026-01-17 10:21:25 +00:00
Evgeny Poberezkin d908404143 Merge branch 'master' into rcv-services 2026-01-15 18:31:09 +00:00
spaced4ndyandEvgeny Poberezkin ca26c69937 db: withSavepoint; agent: correctly handle errors in createWithRandomId for postgres (#1693)
* db: withSavepoint; agent: correctly handle errors in createWithRandomId

* comment

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-01-15 14:59:45 +00:00
Evgeny Poberezkin 58212c421a 6.5.0.7 2026-01-13 20:55:38 +00:00
Evgenyandspaced4ndy 1000107259 agent: clean up old records in batches (#1692)
* agent: clean up old records in batches

* update

* update

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* stabilize tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2026-01-13 19:11:46 +00:00
Evgeny 6aadcf1f3f agent: lock rows for concurrent queries in PostgreSQL (#1688)
* agent: lock rows for concurrent queries in PostgreSQL

* fix race conditions in workers

* refactor
2026-01-08 11:09:58 +00:00
spaced4ndy 07604a146f agent: async command to set connection short link (setConnShortLinkAsync) (#1680) 2026-01-07 15:12:52 +00:00
Ritiek Malhotra 4c782d3191 docker: build multi-arch images for x86-64 and arm64 (#1540) 2026-01-07 10:49:21 +00:00
Evgeny c4b687ba64 agent: reset schema after changes (#1690) 2026-01-05 20:37:50 +00:00
Evgeny a7b43b1a3e agent: use strict tables (#1686)
* agent: use strict tables

* migrate existing tables to strict

* test: verify that all tables are strict

* fix column types for device_token and ntf_mode

* fix encodings and column types for ntf_sub_action and ntf_sub_smp_action

* update schema

* remove debug.trace

* log
2026-01-03 17:19:18 +00:00
Evgeny d6df769799 ntf server: additional tests (#1687) 2026-01-02 14:07:13 +00:00
Evgeny db4b27e88a agent: create user with option to enable client service (#1684)
* agent: create user with option to enable client service

* handle HTTP2 errors

* do not catch async exceptions
2025-12-27 09:12:22 +00:00
Evgeny Poberezkin 9e813c26a7 Merge branch 'master' into rcv-services 2025-12-25 21:52:07 +00:00
Evgeny bafdbc1dec smp protocol: fix encoding for SOKS/ENDS responses (#1683) 2025-12-25 13:00:29 +00:00
Evgeny Poberezkin 5f73d1e629 6.5.0.6 2025-12-23 18:00:26 +00:00
Evgeny Poberezkin 70d1b99fb4 Merge branch 'stable' 2025-12-23 14:18:07 +00:00
Evgeny Poberezkin 2ea98db9d8 6.4.7.0 2025-12-23 12:42:46 +00:00
Evgeny bbe1c716e6 xrcp: treat any 127.x.x.x IP address as local (#1682) 2025-12-23 12:42:05 +00:00
Evgeny Poberezkin 77ac452190 6.5.0.5 2025-12-22 21:56:25 +00:00
Evgeny 11ae20ea20 ntf server: use different client certs for each SMP server, remove support for store log (#1681)
* ntf server: remove support for store log

* ntf server: use different client certificates for each SMP server
2025-12-22 07:56:53 +00:00
Evgeny a1277bf6bf agent: remove service queue association when service ID changed, process ENDS event, test migrating to/from service (#1677)
* agent: remove service queue association when service ID changed

* agent: process ENDS event

* agent: send service subscription error event

* agent: test migrating to/from service subscriptions, fixes

* agent: always remove service when disabled, fix service subscriptions
2025-12-19 21:10:12 +00:00
Evgeny Poberezkin c8a72431c0 Merge branch 'master' into rcv-services 2025-12-15 11:57:46 +00:00
Evgeny ea70575275 agent: use server cert hash when selecting server queues to subscribe (#1679) 2025-12-15 11:48:17 +00:00
Evgeny Poberezkin 568500c6bc Merge branch 'master' into rcv-services 2025-12-14 23:14:18 +00:00
Evgeny 49e9ce1649 agent: skip subscribing servers without queues (#1678) 2025-12-14 23:01:21 +00:00
Evgeny f5eb735551 servers: service stats and logging, allow services without option (removed), report errors during service message delivery, remove threads when service subscription ended (#1676)
* smp server: always allow services without option

* smp server: maintain IDs hash in session subscription states

* smp server: service message delivery error handling

* ntf server: log subscription count and hash differences

* smp server: remove delivery threads when service subscription ended/client disconnected
2025-12-14 12:07:29 +00:00
Evgeny Poberezkin 83894074b0 Merge branch 'master' into rcv-services 2025-12-13 19:08:47 +00:00
Evgeny Poberezkin 2ca440dd2d 6.5.0.4 2025-12-10 21:09:33 +00:00
Evgeny Poberezkin 35fe5ac9e4 Merge branch 'master' into rcv-services 2025-12-05 21:30:28 +00:00
Evgenyandspaced4ndy 2ea9a9a143 agent: finalize initial service subscriptions, remove associations on service ID changes (#1672)
* agent: remove service/queue associations when service ID changes

* agent: check that service ID in NEW response matches session ID in transport session

* agent subscription WIP

* test

* comment

* enable tests

* update queries

* agent: option to add SQLite aggregates to DB connection  (#1673)

* agent: add build_relations_vector function to sqlite

* update aggregate

* use static aggregate

* remove relations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add test, treat BAD_SERVICE as temp error, only remove queue associations on service errors

* add packZipWith for backward compatibility with GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-12-05 20:46:48 +00:00
spaced4ndyandEvgeny Poberezkin 92a9579e69 agent: option to add SQLite aggregates to DB connection (#1673)
* agent: add build_relations_vector function to sqlite

* update aggregate

* use static aggregate

* remove relations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-12-04 08:58:14 +00:00
Evgeny Poberezkin ff7bdbcbeb Merge branch 'master' into rcv-services 2025-12-03 17:23:59 +00:00
Evgeny cf9b7e5b6a agent: option to add SQLite functions to DB connection (#1674)
* agent: option to add SQLite functions to DB connection

* add module
2025-12-03 16:10:35 +00:00
Evgeny 38e899957f agent: service subscription events (#1671)
* agent: use server keyhash when loading service record

* agent: process queue/service associations with delayed subscription results

* agent: service subscription events
2025-11-27 21:37:19 +00:00
Evgeny 5e9b164f4e agent: fail when per-connection transport isolation is used with services (#1670) 2025-11-25 23:17:47 +00:00
Evgeny 3ccf854865 servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1668)
* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)

* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers

* smp server: xor hash with triggers

* fix sql and using pgcrypto extension in tests

* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands

* agent migrations with functions/triggers

* remove agent triggers

* try tracking service subs in the agent (WIP, does not compile)

* Revert "try tracking service subs in the agent (WIP, does not compile)"

This reverts commit 59e908100d.

* comment

* agent database triggers

* service subscriptions in the client

* test / fix client services

* update schema

* fix postgres migration

* update schema

* move schema test to the end

* use static function with SQLite to avoid dynamic wrapper
2025-11-25 16:55:59 +00:00
Evgeny 1ca4677b28 smp server: messaging services (#1565)
* smp server: refactor message delivery to always respond SOK to subscriptions

* refactor ntf subscribe

* cancel subscription thread and reduce service subscription count when queue is deleted

* subscribe rcv service, deliver sent messages to subscribed service

* subscribe rcv service to messages (TODO delivery on subscription)

* WIP

* efficient initial delivery of messages to subscribed service

* test: delivery to client with service certificate

* test: upgrade/downgrade to/from service subscriptions

* remove service association from agent API, add per-user flag to use the service

* agent client (WIP)

* service certificates in the client

* rfc about drift detection, and SALL to mark end of message delivery

* fix test

* fix test

* add function for postgresql message storage

* update migration
2025-11-07 21:36:28 +00:00
spaced4ndyandEvgeny Poberezkin 3016b929b4 agent: make createConnection and setConnShortLink apis support setting all link data fields (#1665)
* agent: make createConnection and setConnShortLink apis support setting all link data fields

* add functions

* refactor

* refactor

* fix tests

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-11-05 07:08:33 +00:00
Evgeny Poberezkin 1ae3e8d0be 6.5.0.3 2025-10-23 08:34:11 +01:00
Evgeny 1329fc726f smp: support client notices (#1659)
* agent: support client notices

* improve

* fix, test

* rename

* cleanup

* send and process notices in more cases

* dont delete

* dont remove notice on other permanent errors

* dont remove notice if there is no notice ID in queue

* add server to error

* allow deleting

* only use notice if key hash matches
2025-10-17 18:34:59 +01:00
spaced4ndyandEvgeny 234aeb81dd agent: add subscription status to connection stats (#1658)
* agent: add subscription status to connection stats

* wip

* conn status

* format

* refactor

* refactor

* m

* shorter

* shorter

---------

Co-authored-by: Evgeny <evgeny@poberezkin.com>
2025-10-13 17:53:28 +01:00
spaced4ndyandEvgeny Poberezkin 1dbc15b2e6 agent: sync connections (#1654)
* agent: sync subscriptions

* remove comment

* add shouldDelete flag

* compare api

* remove instance

* query

* rename

* refactor

* functor

* JSON instances

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-10-10 05:41:42 +00:00
Evgeny 80aa56cbcc agent: track which queues need subscribing for iOS NSE (#1657)
* agent: track which queues need subscribing for iOS NSE

* fix down migration

* fix, cleanup
2025-10-09 18:33:27 +01:00
Evgeny 9cda20381f agent: subscribe all connections (#1655)
* agent: subscribe all connections

* query, version

* BoolInt

* add query to errors

* Revert "add query to errors"

This reverts commit 32a1f7fe11.

* fix optional field

* version

* limit number of in-flight subscriptions to 35000
2025-10-09 13:43:48 +01:00
Evgeny 318ddf692a smp server: import/export to/from PostgreSQL with one command, deprecation notice for journal storage (#1653) 2025-10-09 13:31:57 +01:00
Evgeny 11a4859f8e agent: batch processing of subscription results and errors (#1652)
* agent: batch processing of subscription results and errors

* run agent tests with in-memory server storage

* version

* non empty errors

* size
2025-10-07 14:49:38 +01:00
Evgeny a3d1a72eb0 agent: optimize subscriptions memory usage (#1651)
* agent: optimize subscriptions memory usage more (do not store subscribed queues in memory) WIP

* use new session subscriptions data

* version

* remove old data structure

* remove version

* batch deletions

* test TSessionSubs

* comment
2025-10-06 15:21:17 +01:00
Evgeny Poberezkin 17b71cf673 Merge branch 'stable' 2025-10-05 22:03:58 +01:00
Evgeny 779222d1a7 agent: optimize subscriptions memory usage (#1649)
* agent: optimize subscriptions memory usage

* version

* remove typeclass

* disable subscription to test memory usage

* disable delivery and commands

* disable reading connections

* enable subscriptions

* RcvQueueCred

* RcvQueueCred postgres

* fix

* optimize keys

* version

* rename

* comments
2025-10-05 13:35:54 +01:00
Evgeny Poberezkin 538dcb6a4c 6.4.6.0 2025-10-04 14:49:42 +01:00
Evgeny Poberezkin 98115b6db1 6.5.0.2 2025-09-30 12:37:28 +01:00
Evgeny 9854caca33 agent: optimize subscriptions (#1645)
* agent: optimize subscriptions

* simplify

* clean up
2025-09-30 12:34:42 +01:00
spaced4ndy c8b551dcf7 agent: fix file reception error in postgres (#1647) 2025-09-30 08:32:45 +01:00
Evgeny 99f40ae109 support for additional database migrations (#1644) 2025-09-27 11:21:38 +01:00
sh 3a9381b966 ci/scripts: execute git version if chmod fails (#1643) 2025-09-23 09:08:56 +01:00
Evgeny Poberezkin 54a2a6c905 6.5.0.1 2025-09-17 09:04:03 +01:00
Evgeny 7b7616ce7e agent: batch loading of connections with PostgreSQL client (#1639)
* agent: batch loading of connections with PostgreSQL client

* batch more

* optimize getPendingCommandServers

* fix Bool conversion

* enable all tests

* cleanup
2025-09-16 14:28:06 +01:00
sh b020a08ea0 ci: disable aarch64 tests (#1638) 2025-09-15 09:02:34 +01:00
Evgeny Poberezkin 2737e7d602 6.5.0.0.1 2025-09-15 07:52:17 +01:00
Evgeny d3754b36bc smp server: do not use queue cache with PostgreSQL message storage (#1637)
* smp server: do not use queue cache with PostgreSQL message storage

* fix loading queues via notifier IDs
2025-09-15 07:51:27 +01:00
Evgeny Poberezkin 112cd9d5f4 Merge branch 'stable' 2025-09-14 14:42:05 +01:00
Evgeny 8a7991a376 agent: option to set SQLite database backup path (#1636)
* agent: option to set SQLite database backup path

* fix test compilation
2025-09-14 12:41:49 +01:00
Evgeny 6f4d0f5377 Merge pull request #1630 from simplex-chat/db-messages
Feature branch: PostgreSQL message store
2025-09-13 19:24:59 +01:00
Evgeny 9cfdae3494 smp server: uniterruptible mask for DB operations (#1635) 2025-09-13 19:08:07 +01:00
Evgeny Poberezkin 62183699db 6.5.0.0 2025-09-13 11:37:59 +01:00
Evgeny 9be9a76b42 smp server: improve message expiration (#1634)
* smp server: limit by time the queues to export journal messages for

* pass queue/msg thresholds separately

* reset db connection on errors

* Revert "smp server: limit by time the queues to export journal messages for"

This reverts commit d3bc0cba4b.

* fix test compilation

* flag to expire messages

* improve test

* expire messages newer than quota
2025-09-13 11:35:34 +01:00
Evgeny Poberezkin 80f7be6dd7 Merge branch 'master' into db-messages 2025-09-13 09:31:23 +01:00
Evgeny 83721240a4 reset db connection on errors (#1633) 2025-09-13 09:30:01 +01:00
Evgeny 6c66cf367a smp server: set message counts correctly after import (#1632) 2025-09-12 13:08:11 +01:00
Evgeny a137d01c90 smp server: improve counts management and expiration for postgres message store (#1631)
* smp server: improve counts management and expiration for postgres message store

* one update

* fix expiration
2025-09-12 11:39:35 +01:00
Evgeny bac6ea6e91 smp server: store messages in PostgreSQL (#1622)
* smp server: store messages in PostgreSQL

* stored procedures to write and to expire messages

* function to export messages

* move all message functions to PostgreSQL, remove delete trigger

* comments

* import messages to db

* fix message import, add export

* fix export

* fix export

* fix compilation flags

* import messages line by line

* fix server start with database storage

* fix compilation

* comments
2025-09-11 20:22:55 +01:00
Evgeny 0c1030cf02 smp server: faster export (#1626)
* smp server: faster export

* flush

* export messages with PostgreSQL database

* remove flush

* batch message writes

* Revert "batch message writes"

This reverts commit 61fb3c3b9e.

* remove $!

* fast journal export

* another approach

* Revert "smp server: remove dependency of message size on the version (#1627)"

This reverts commit 8fea15245a.

* style

* faster?

* cleanup

* cleanup

* refactor

* refactor

* concurrent read messages

* Revert "concurrent read messages"

This reverts commit 05a32e6491.

* concurrent read/write

* parameter to export to/import from another message log file

* Revert "parameter to export to/import from another message log file"

This reverts commit 4e88b035c4.

* Revert "concurrent read/write"

This reverts commit a8eab1f7e3.
2025-09-10 18:22:58 +01:00
Evgeny Poberezkin 23aff6b0b1 Revert "smp server: remove dependency of message size on the version (#1627)"
This reverts commit 8fea15245a.
2025-09-09 09:56:48 +01:00
sh 3335cd5500 ci: add aarch64 builds (#1624) 2025-09-09 08:55:37 +01:00
spaced4ndyandEvgeny Poberezkin a4f049d8da agent: parameterize withWork, getWorkItem with StoreError; parameterized Binary for SQLite (#1617)
* agent: parameterize withWork StoreError

* getWorkItem

* export

* binary

* remove handleWrkErr AnyStoreError constraint

* put AnyError in AnyStoreError constraint

* move typeclass

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-09-08 16:38:08 +01:00
Evgeny 8fea15245a smp server: remove dependency of message size on the version (#1627) 2025-09-08 11:28:42 +01:00
sh 42a2c1fc57 ci: fix git in docker (#1623) 2025-09-08 09:41:20 +01:00
Evgeny Poberezkin 7e98b3103f 6.4.5.2 2025-09-07 15:13:19 +01:00
Evgeny Poberezkin 2a61085f07 backward compatible encoding for missing NetworkError 2025-09-07 15:04:06 +01:00
Evgeny 4386dd8b5a smp server: option to expire messages when new message is sent (now off by default) (#1625)
* smp server: option to expire messages when new message is sent (now off by default)

* fix test
2025-09-07 09:02:32 +01:00
Evgeny Poberezkin 50ddd630be 6.4.5.1 2025-09-03 11:12:20 +01:00
Evgeny cb3250e7b4 servers: better socket leak prevention during TLS handshake, add NetworkError type to better diagnose connection errors (#1619)
* servers: better socket leak prevention during TLS handshake

* log tcp connection errors

* more detailed network error

* log full address

* rename error

* add encodings for NetworkError

* refactor

* comment

* bind

* style

* remove parameters of NETWORK error from encoding
2025-09-02 16:07:37 +01:00
Evgeny Poberezkin 0319addd2b 6.4.5.0 2025-09-01 18:57:09 +01:00
Evgeny 77bf76e1f9 smp server: fix in-memory server not restoring queue/service associations after 2+ restarts (#1618)
* smp server: fix in-memory server not restoring queue/service associations after 2+ restarts

* fix test, do not reuse database
2025-09-01 17:45:31 +01:00
Evgeny beafac1f73 agent: make agent workers usable from other contexts (#1614) 2025-08-29 08:33:55 +01:00
Evgeny Poberezkin a2d777bda0 6.4.4.1 2025-08-23 20:05:10 +01:00
Evgeny e48bedeaf2 servers: fix server pages when source code is not specified, include commit SHA in printed version and in web page. (#1608)
* smp server: fix server pages when source code is not specified

* servers: include git commit in version

* flexible alpn

* fix test

* fix ghc 8.10.7 build
2025-08-23 19:59:00 +01:00
Evgeny Poberezkin a2d35281b2 6.4.4.0 2025-08-18 16:22:24 +01:00
spaced4ndy 46035af9a3 agent: add context to SEMsgNotFound error (#1604) 2025-08-18 10:59:57 +01:00
Evgeny Poberezkin 4b7fc34fe3 enable all tests 2025-08-17 10:40:12 +01:00
Evgeny 96e8b4a146 smp server: wrap all queries in transactions (#1603)
* smp server: wrap all queries in transactions

* fix test

* fix schema test
2025-08-17 10:38:20 +01:00
Evgeny 2cedb66667 agent: chat relay address type for short links (#1602) 2025-08-15 07:24:37 +01:00
Evgeny e345671c76 agent: extend xrcp certificate validity 1 hour in the past, to allow out of sync clocks (#1601) 2025-08-14 22:48:11 +01:00
Evgeny 86fb2cddc5 smp server: support short SimpleX addresses in server information / pages (#1600) 2025-08-12 07:31:32 +01:00
Evgeny Poberezkin 931c533a3d 6.4.3.0 2025-08-05 12:30:28 +01:00
Evgeny 79ba60e3ad agent: update contact request server hosts to match server in short link (#1597) 2025-08-05 12:29:50 +01:00
Evgeny fb477b24d7 smp server: support short link URI as queue identifier in control port commands (#1596) 2025-08-05 08:18:39 +01:00
Evgeny Poberezkin 9f263e8f3e docs: changelog 2025-08-02 09:21:44 +01:00
Evgeny Poberezkin db325cb81f 6.4.2.0 2025-07-31 09:21:11 +01:00
Evgeny b167d01f8a rfc: using the same profile from multiple devices (#1588) 2025-07-30 20:54:03 +01:00
Evgenyandsh f4e7469f96 xftp server: prometheus metrics (#1595)
* xftp server: prometheus metrics

* update

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-07-30 16:18:49 +01:00
Evgeny 4647d69d4b smp server: do not include previously blocked queues in stats, prevent leak of client threads (#1593) 2025-07-30 11:12:57 +01:00
sh 9ab071d62c ci: fix reproduce-schedule (#1594) 2025-07-30 09:17:28 +01:00
Evgeny Poberezkin f4c09ac51f Merge branch 'stable' 2025-07-29 09:09:36 +01:00
Evgeny Poberezkin 846be50f72 6.4.1.0 2025-07-21 15:22:47 +01:00
Evgeny c0f357d817 servers: control port session improvements (#1591)
* servers: prohibit changing role during control port session

* quota for blocked queues

* allow disabling blocking and quota

* fix test

* fix INI file
2025-07-21 15:19:43 +01:00
Evgeny 40fc09a93d smp server: additional stat counter for ntf credentials created together with the queue (#1589)
* smp server: additional stat counter for ntf credentials created together with the queue

* fix prometheus

* fix test

* fix qSub
2025-07-18 09:26:25 +01:00
Evgeny 2a90a2c552 smp protocol: create notification credentials via NEW command that creates the queue (#1586)
* smp protocol: create notification credentials via NEW command that creates the queue

* create ntf subscription for queues created with ntf credetials

* do not create ntf credentials when switching connection to another queue
2025-07-16 16:46:45 +01:00
sh fc581bf729 github/workflows: fix reproduce schedule (#1587) 2025-07-16 11:42:14 +01:00
Evgeny Poberezkin b6ea025333 docs: update build badge 2025-07-15 10:03:27 +01:00
Evgeny Poberezkin d3e2d9ac5b docs: changelog v6.4 2025-07-13 15:20:26 +01:00
Evgeny Poberezkin 85a7fbf538 6.4.0.12 2025-07-12 14:19:32 +01:00
Evgeny 62733ef4c1 smp server: refactor subscriptions and delivery in order to always response SOK on subscription with an optional message to follow. (#1573)
* smp server: refactor subscriptions and delivery

* metric for time between MSG and ACK

* cleanup

* refactor pattern match for ghc 8.10.7

* time buckets

* split max time metric

* histogram

* fix
2025-07-12 14:18:38 +01:00
Evgeny 99e59b73a3 agent: fix message delivery in case one of the connections has no snd queue for any reason - it could break delivery to all connections (#1585)
* agent: fix message delivery in case one of the connections has no snd queue for any reason - it could break delivery to all connections

* simplify

* comment

* refactor, also postpone failing on ratchet sync send prohibited errors

* postpone failing on connection errors to allow subsequent connections succeed
2025-07-10 10:21:12 +01:00
Evgeny 384a3352cf ntf server: more efficient status update query (#1584) 2025-07-08 23:01:57 +01:00
Evgeny Poberezkin 7a04fdff2e 6.4.0.11.3 2025-07-07 18:50:31 +01:00
Evgeny ba3c75e58c smp server: correctly track if ntf service is subscribed and total subscribed queues count (fixes race condition between NSUB and NSUBS from notification server) (#1583) 2025-07-07 18:49:47 +01:00
Evgeny Poberezkin 1062ccc5c3 6.4.0.11.2 2025-07-07 10:07:54 +01:00
Evgeny 36f05e272e agent: support different timeouts for interactive and background requests (#1582)
* agent: support different timeouts for interactive and background requests

* fix tests

* use one constructor for the first request and for retries
2025-07-07 09:38:52 +01:00
Evgeny Poberezkin 660c4293f0 6.4.0.11.1 2025-07-05 11:35:27 +01:00
spaced4ndyandEvgeny Poberezkin 1b8613d767 build: do not exclude postgres modules for client_library flag (#1579)
* don't exclude postgres modules for client_library flag

* fix build with client and server postgres flag

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-07-04 17:31:57 +01:00
Evgeny bfa52c4ba5 docs: update RFC for short links to match the implementation (#1581) 2025-07-03 18:09:02 +01:00
spaced4ndy c5eb66038b agent: allow to accept contact requests after address is deleted (#1580) 2025-07-03 09:29:39 +00:00
Evgeny Poberezkin a46edd60f0 6.4.0.11 2025-06-30 21:50:21 +01:00
Evgeny b4bcfd325b agent: better handling errors during connection handshake retries (#1578)
* agent: handle invitation connection handshake errors

* fix/test retries for connecting via address
2025-06-25 19:06:00 +01:00
Evgeny 976bd3a389 agent: short link encodings (#1577) 2025-06-24 10:06:44 +01:00
Evgeny Poberezkin bf27c846da 6.4.0.10.1 2025-06-20 13:49:17 +01:00
Evgeny 455360205c smp server: prometheus histogram for message confirmation times (ACK) (#1575)
* time buckets

* split max time metric

* histogram

* histogram for confirmed delivery times

* gaugehistogram

* fix created, _ in gauge_histogram

* remove comments

* fix metrics
2025-06-20 13:46:59 +01:00
Evgeny 79c67f2026 agent: JSON encoding for connection mode (#1576) 2025-06-19 11:21:29 +01:00
Evgeny c8928626fc ntf server: do not log subscription status updates on subscribing/unsubscribing (#1574) 2025-06-19 09:15:07 +01:00
Evgeny b47d28a22a smp server: prometheus metrics for delievered and not acknowledged messages (to monitor stuck deliveries) (#1572) 2025-06-17 21:10:09 +01:00
Evgeny c5b7d3c7af agent: use flexible (small/large) string encoding for contact link data, for backward/forward compatibility (#1569)
* agent: use large string encoding for contact link data

* size-dependent user link data encoding

* tag large user data encoding with \255
2025-06-16 11:11:59 +01:00
Evgeny Poberezkin d950012530 6.4.0.10 2025-06-14 21:06:54 +01:00
Evgeny Poberezkin 27b1f48929 Merge branch 'stable' 2025-06-14 21:05:59 +01:00
Evgeny Poberezkin 3d62a383d5 6.4.0.3.1 2025-06-14 14:19:31 +01:00
Evgeny 6ac7101f4f agent: make decoding for short link data forward compatible (#1568) 2025-06-14 14:13:14 +01:00
Evgeny 65cc19842c servers: restore constant time for absent queue/wrong key (regression in 6.4-beta.9) (#1567) 2025-06-14 07:10:51 +01:00
Evgeny 656f290660 agent: fix updating short invitation link data (#1566) 2025-06-13 21:27:46 +01:00
Evgeny Poberezkin 643c3c3b3e 6.4.0.9 2025-06-13 09:48:20 +01:00
Evgeny da37384335 smp server: batch commands (#1559)
* protocol: refactor types and encoding

* clean

* smp server: batch commands (#1560)

* smp server: batch commands verification into one DB transaction

* ghc 8.10.7

* flatten transmission tuples

* diff

* only use batch logic if there is more than one transmission

* func

* reset NTF service when adding notifier

* version

* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"

This reverts commit 3df2425162.

* version

* Revert "version"

This reverts commit d80a6b74c5.
2025-06-12 23:05:04 +01:00
Evgeny 1658048c2c Revert "smp server: use separate database pool for reading queues and creating service records (#1561)" (#1564)
This reverts commit 3df2425162.
2025-06-12 11:04:43 +01:00
Evgeny 27d38518e1 agent: use PQ keys in contact request data inside link container (but not in contact request link); use PQ keys in invitations sent to contact addresses (#1563)
* agent: use PQ keys in contact request data inside link container (but not in contact request link); use PQ keys in invitations sent to contact addresses

* do not use PQ keys in the link with old address versions
2025-06-11 18:22:37 +01:00
Evgeny cf8088ac6a smp server: do not require ALPN for HTTP to fix Android asset links, do not use port 443 in server-server SMP connections (#1562)
* log alpn

* always use HTTP when SNI is sent, regardless of ALPN

* decide credential based on SNI

* do not use web port in SMP/Ntf servers connecting to SMP servers

* simpler

* refactor

* fix
2025-06-10 22:09:07 +01:00
Evgeny Poberezkin 1e82104224 6.4.0.8 2025-06-09 14:20:25 +01:00
Evgeny 46ff37c362 ntf server: additional statistics (#1558)
* ntf server: additional statistics

* version

* fix stats

* add stats to track notifications without active token

* refactor

* fix stats parser

* version
2025-06-09 14:14:27 +01:00
Evgeny 3df2425162 smp server: use separate database pool for reading queues and creating service records (#1561) 2025-06-09 12:47:35 +01:00
Evgeny 5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny 8e86c97a13 servers: include supported ALPNs in server transport config (#1557) 2025-06-03 13:35:35 +01:00
Evgeny 90e8c3adf6 smp agent: use static RNG function to avoid dynamic C stub created by Haskell FFI wrapper (#1556) 2025-06-02 20:11:27 +01:00
Evgeny 56851365b1 servers: parameterize SMP agent to optimize subscriptions (#1555)
* servers: parameterize SMP agent to optimize subscriptions

* refactor ClientCommand to avoid additional map in the notification server agent

* rename
2025-06-01 16:14:52 +01:00
Evgeny a9814bb6d3 smp server: parameterize server by store type to avoid boxing/unboxing client handles (#1554)
* smp server: parameterize server by store type to avoid boxing/unboxing client handles

* ghc 8.10.7
2025-05-29 22:17:28 +01:00
Evgeny Poberezkin 3ad8bd15a6 6.4.0.7 2025-05-29 13:14:09 +01:00
spaced4ndy 4c33d8ac43 agent: To and FromField instances for AConnShortLink (#1553) 2025-05-29 10:59:36 +01:00
spaced4ndyandEvgeny Poberezkin a94ca62624 agent: allow changing invitation link data before it is secured (#1552)
* agent: setInvitationShortLink api

* Eq instance

* allow changing link data on server, refactor

* fix

* encodings

* remove link data after connection

* Revert "encodings"

This reverts commit f8e254cca9.

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-05-29 10:44:16 +01:00
Evgeny 53b72469b6 refactor types (#1551) 2025-05-27 22:31:02 +01:00
Evgeny Poberezkin f80ed32a06 6.4.0.6 2025-05-25 17:28:39 +01:00
Evgeny 07eaf9157b smp server: allow getting and deleting short links for the old contact queues (#1549)
* smp server: allow getting and deleting short links for the old contact queues

* fix verifaction of legacy contact queues

* test
2025-05-25 17:03:02 +01:00
Evgeny 56ea2fdd56 refactor types for DB entity (#1548) 2025-05-24 18:19:11 +01:00
Evgeny ffecd4a17a parameterize transport by peer type (client/server) (#1545)
* parameterize transport by peer type (client/server)

* LogDebug level when test is retried

* support "flipped" HTTP2, fix test retry to avoid retrying pending tests

* move sync to the end of the tests
2025-05-24 14:34:22 +01:00
Evgeny Poberezkin dae649fb87 6.4.0.5 2025-05-24 14:21:53 +01:00
Evgeny 57a77f75c1 smp server: support adding link data to contact addresses created before July 2024 (#1547) 2025-05-24 14:20:25 +01:00
spaced4ndy 18e73b8aa7 agent: pass CRClientData to setContactShortLink (#1546)
* agent: pass CRClientData to setContactShortLink

* fix

* fix
2025-05-23 18:21:36 +01:00
Evgeny af9ca59e51 smp server: optimize concurrency and memory usage, refactor (#1544)
* smp server: optimize concurrency and memory usage, refactor

* hide clients IntMap

* reduce STM contention

* comment

* version

* correct stats for subscriptions

* version

* comment

* remove subscribed clients from map

* version

* optimze, refactor

* version

* debug test

* enable all tests

* remove test logs

* retry failed tests with debug logging

* increase test timeout

* sync between tests
2025-05-23 12:52:18 +01:00
Evgeny Poberezkin d352d518c2 6.4.0.4 2025-05-21 08:56:01 +01:00
Evgeny f0dc600016 ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log
2025-05-21 08:55:32 +01:00
Evgeny Poberezkin f44ea0a6d8 6.4.0.3 2025-05-15 21:01:49 +01:00
Evgeny f7d31d4c02 Merge pull request #1524 from simplex-chat/ntf-storage
ntf server: PostgreSQL database storage (feature branch)
2025-05-15 21:00:58 +01:00
Evgeny b90e25a3a5 ntf server: fix repeat token registration when it became invalid (regression) (#1539) 2025-05-15 19:56:48 +01:00
Evgeny Poberezkin cf4b9f669d agent: always enable periodic notifications 2025-05-15 15:16:44 +01:00
Evgeny Poberezkin e417d35cce Merge branch 'master' into ntf-storage 2025-05-14 22:20:55 +01:00
Evgeny Poberezkin deaec3cce2 6.4.0.2 2025-05-07 07:24:06 +01:00
Evgeny 7bbd99644a agent: fix query that updates last message timestamps for notification processing (#1536) 2025-05-06 23:19:58 +01:00
Evgeny cb59a449dd agent: return error and message absence differently when getting notification messages (#1535)
* agent: return error and message absence differently when getting notification messages

* fix test

* mapM

* inline nse functions, release lock on error or no message
2025-05-06 16:20:01 +01:00
Evgeny a632eea75b protocol: parsing for UTCTime to stop on "," and ";" (#1532) 2025-05-04 21:11:06 +01:00
Evgeny 3d10c9bf9e agent: handle cases when last message ts is not set for notifications; set last ts for "stale" notifications when messages expired and queue is empty, to prevent repeated processing (#1531)
* agent: handle cases when last message ts is not set for notifications; set last ts for "stale" notifications when messages expired and queue is empty, to prevent repeated processing

* only log errors if they exist

* only set last ts for queue that delivered notification
2025-05-02 11:21:39 +01:00
Evgeny 2f0cdc40af ntf server: fix returned last notifications (#1530) 2025-04-30 14:57:04 +01:00
Evgeny 0a3d014f5d ntf server: priority db pool for notifications and client commands, concurrent resubscriptions, CLI options (#1529)
* ntf server: option to skip specific tokens when importing store log

* logs

* batch later

* subscribe in parallel, bigger queues

* ntf server: test schema and migrations

* optimize subscriptions

* log level

* resubscribe before starting

* better subscribing

* pooled concurrent resubscriptions

* 5

* async resubscribe

* typo

* priority pool

* db batch size
2025-04-30 11:40:32 +01:00
Evgeny 7d0115daec ntf server, agent: send all periodic notifications from one thread, only to old active clients or new clients with periodic notification mode (#1528)
* ntf server, agent: send all periodic notifications from one thread, only to old active clients or new clients with periodic notification mode

* send different type via subscription queues

* option to compact store log on start
2025-04-28 12:18:55 +01:00
Evgeny f024ab1c3f ntf server: prometheus metrics (#1527)
* ntf server: save prometheus stats

* info metrics

* fix test
2025-04-27 13:25:44 +01:00
Evgeny f4bc1f0926 ntf server: skip duplicates when importing tokens and subscriptions (#1526)
* ntf server: skip duplicates when importing tokens and subscriptions

* skip imported last notifications when no token or subscription present

* fix skipped imported notifications count

* all tests

* fix test
2025-04-26 23:37:55 +01:00
Evgeny 42dbb887f7 ntf server: allow retries when creating subscriptions, prohibit subscriptions with the same queue but another notifier key or token (#1525)
* ntf server: allow retries when creating subscriptions, prohibit subscriptions with the same queue but another notifier key or token

* sync files in the test

* refactor
2025-04-25 19:41:18 +01:00
Evgeny 850d2fa423 ntf server: PostgreSQL database storage (#1519)
* ntf server: PostgreSQL database storage

* ntf server: import/export stubs

* ntf server postgres db functions

* some notifications tests pass

* notifications tests pass

* import/export notification store logs

* fix ntf server CLI

* log in parralel

* update subscription statuses using executeMany

* fix import/export

* refactor

* fix queries

* prohibit token_id and subscription_id updates, dont update tokens and subscriptions on conflict, improve server insertion, remove duplicate tokens for import, remove subscriptions without tokens for import

* comment

* remame

* increase test delay
2025-04-25 16:47:39 +01:00
Evgeny 08b84deba4 agent: option to use web port by default for preset servers only (#1523)
* agent: option to use web port by default for preset servers only

* shorten/restore short links in agent, add encodings for SMP web port setting

* decouple preset domains from preset servers for short links

* refactor, rename
2025-04-24 17:11:52 +01:00
Evgeny Poberezkin ec5a60430d 6.4.0.1 2025-04-22 22:01:16 +01:00
Evgeny e4d4b51c59 smp server: fix/test database import (#1521) 2025-04-22 21:59:27 +01:00
Evgeny Poberezkin 6b60f8bab6 Revert "ntf server: optimize in-memory storage (#1516)"
This reverts commit afb338a41a.
2025-04-21 21:01:24 +01:00
shum aa9b93eee5 ci: retry only in stable or release 2025-04-21 19:06:04 +01:00
Evgeny afb338a41a ntf server: optimize in-memory storage (#1516)
* ntf server: optimize in-memory storage

* test

* ntf server: fix store log parser for token status
2025-04-21 17:12:16 +01:00
Evgenyandshum 1e29f7c811 ci: adjust output, retry tests on failure, include hashes in release notes (#1520)
* scripts/reproduce-builds: rename with prefix and adjust output

* ci: rerun tests and include hashes in release

* ci: fix variables and really skip 8.10.7

* ci: fix prepare step

* ci: setup swap for 8.10.7

* ci: fix upload and formatting

* ci: properly format hashes

* ci: simplify

---------

Co-authored-by: shum <shum@liber.li>
2025-04-21 17:05:31 +01:00
Evgeny Poberezkin d11be15295 ntf server: fix store log parser for token status 2025-04-16 13:01:14 +01:00
Evgeny Poberezkin 305f79d2a6 6.4.0.0 2025-04-14 10:52:05 +01:00
Evgeny Poberezkin af988f774e Merge branch 'stable' 2025-04-14 10:51:34 +01:00
Evgeny Poberezkin 82e389298a docs: CHANGELOG 2025-04-14 10:51:23 +01:00
Evgeny c784d5ce0c Merge pull request #1495 from simplex-chat/short-links
smp protocol: short 1-time invitations and contact address links
2025-04-14 09:58:09 +01:00
Evgeny c2c4730953 smp server: serve .well-known folder via server pages (#1514) 2025-04-13 22:02:48 +01:00
Evgeny Poberezkin ccdd8e1775 agent: additional encodings for short links 2025-04-13 13:43:27 +01:00
Evgeny Poberezkin 307a784174 Merge branch 'master' into short-links 2025-04-13 10:51:32 +01:00
Evgeny Poberezkin 9abc0fa88d 6.3.2.0 2025-04-11 23:22:48 +01:00
Evgeny 76aad61f00 smp server: update server page for short links to correctly change URI for apps (#1511)
* smp server: update server page for short links to correctly change URI for apps

* change page titles
2025-04-11 23:18:23 +01:00
Evgeny 37ce109009 smp server: .well-known folder for server pages for SimpleX apps to handle preset server links (#1510)
* smp server: .well-known folder for server pages for SimpleX apps to handle preset server links

* add short link group path

* test
2025-04-11 18:30:43 +01:00
Evgeny a66163dc46 agent: additional short link types and encodings (#1509) 2025-04-11 12:29:44 +01:00
Evgeny Poberezkin bdf8bf093c smp server: make symlinks relative 2025-04-10 20:58:33 +01:00
Evgenyandsh b7a9542213 smp server: short links and owners for channels (#1506)
* smp server: short links and owners for channels

* types

* support mutliple rcv keys

* fix down migration, test/create server schema dump

* reduce schema dump

* parameterize type for link data by connection type

* return full connection link data

* test version

* change short link encoding

* test: print pg_dump output

* server pages, link encoding

* fix connection request when queue data and sender ID are created for old servers

* test, change pattern

* ci: install postgresql tools in runner (#1507)

* ci: install postgresql tools in runner

* ci: docker shell abort on error

* fix pattern for ghc 8.10.7

* patch ConnReqUriData SMP encoding to preserve queue mode after decoding

* test for RKEY

* fix/test store log with RKEY

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-04-10 19:09:47 +01:00
Evgeny Poberezkin 3a3f9fd51e Merge branch 'master' into short-links 2025-04-10 11:51:37 +01:00
Evgeny 6dc9d76ed3 smp agent: handle client/agent version downgrades after connection was established (#1508) 2025-04-10 11:50:19 +01:00
Evgeny a2a4b80af4 agent: padded encryption for link data, shorten/restore short links, tests (#1505)
* agent: padded encryption for link data, tests

* lambda

* test short links via proxy

* tests: server persistence with short links

* rfc: group links

* shorten, restore, test short links encoding

* rfc
2025-04-05 07:45:39 +01:00
Evgeny 7ec0ae3bb5 agent: types and encodings to use from the app (#1504)
* agent: types and encodings to use from the app

* use action forks
2025-04-03 16:31:22 +01:00
Evgeny 2c5530c9f0 agent: binary protocol encoding for connection request (#1503)
* agent: binary protocol encoding for connection request

* enable tests

* test
2025-04-01 22:32:49 +01:00
Evgeny 94ee3ceced agent: replace sndSecure with queueMode in queue URIs and records (#1502)
* agent: replace sndSecure with queueMode in queue URIs and records

* fix test
2025-03-31 16:54:29 +01:00
Evgeny Poberezkin dd67de4d71 Merge branch 'master' into short-links 2025-03-31 16:54:07 +01:00
Evgeny 285fd93c32 smp server: notices about server pages and port 443, enable store log by default (#1501)
* smp server: notices about server pages and port 443, enable store log by default

* enable store log by default for ntf and xftp servers, fix tests
2025-03-31 16:41:11 +01:00
Evgeny 56bec06856 smp protocol: remove creating notifications from NEW command, new ideas for notifications (#1500) 2025-03-30 19:56:16 +01:00
Evgeny 04cbed90fb agent: set/update and delete contact short link data (#1499)
* agent: set/update and delete contact short link data

* delete contact link data, tests

* comments

* type signature for GHC 8.10.7
2025-03-30 11:17:25 +01:00
Evgeny c1a6647f19 agent: join connection when 1-time invitation short link is already secured (#1496)
* agent: join connection when 1-time invitation short link is already secured

* do not pass short link to join

* delete short link record after connection
2025-03-29 09:42:56 +00:00
Evgeny Poberezkin 1dd677eec2 Merge branch 'master' into short-links 2025-03-28 19:50:18 +00:00
Evgeny 7636bc7491 smp server: remove locks for deleted queues, additional statistics for objects in memory (#1498)
* smp server: remove locks for deleted queues, additional statistics for objects in memory

* version

* reduce queue cache usage

* less caching, refactor

* comments

* revert version
2025-03-28 18:51:54 +00:00
sh 79adb83782 ci: add reproduce builds workflow (#1497)
* ci: add reproduce builds workflow

* ci: trigger webhook and build every day

* ci: change secrets
2025-03-27 16:06:32 +00:00
Evgeny b83d897650 smp protocol: short links and other changes from RFC (#1489)
* smp protocol: short links types and other changes from RFC

* add fields for queue link ID and data

* create queue and ntf credentials with NEW command

* all tests

* simplfiy types, update rfc

* update rfc

* include SenderId in NEW request in case queue data is sent

* store queue data and generate link ID if needed

* update rfc

* agent API and types

* SMP commands and persistence for short links

* SMP client functions for short links

* agent client functions for short links

* create rcv queue with short link (TODO secret_box)

* encryption and encoding for link data, postgres client migration

* test creating short link

* get link and data, tests

* comments

* type signature
2025-03-26 17:26:27 +00:00
Evgeny Poberezkin 0c3b25706a Merge branch 'stable' 2025-03-23 18:18:07 +00:00
sh c3f57beafd scripts: fix script for reproducible builds (#1492)
* scripts: fix script for reproducible builds

* scripts: add automatic hash comparasion + compute file with checksums

* scripts: silence cleanup

* ci/scripts: enable parallel build (#1493)
2025-03-23 18:17:27 +00:00
Evgeny Poberezkin aace3fd2fb 6.3.1.0 2025-03-22 13:51:55 +00:00
sh 2e67ed9c4c ci/scripts: refactor reproducible builds (#1490)
* ci: build cache, remove 20.04, minor refactor

* scripts: update build reproducibility script

* script: add postgresql reproducibility

* script: no cache when building image

* scripts: force symlinks

* scripts: also download prebuilt postgresql binary

* scripts: add missing app var
2025-03-22 13:27:11 +00:00
Evgeny 614fa2b163 smp server: reduce queue expiration/idle intervals, skip expiring very old queues (#1488)
* smp server: reduce idle queue interval and queue expiration interval

* only expire recent queues (not very old)

* fix

* version
2025-03-20 08:57:47 +00:00
Evgeny 903e96bdfa rfc: smp queue protocol changes (#1485)
* rfc: smp queue protocol changes

* questions

* update

* update

* correction
2025-03-19 10:47:57 +00:00
Evgeny 5c0adcbbff Merge pull request #1467 from simplex-chat/postgres
SMP server PostgreSQL storage - feature branch (merge, dont squash)
2025-03-18 16:48:25 +00:00
sh b2f16eeff4 ci: seperate builds (pg/standard) (#1486) 2025-03-18 09:42:53 +00:00
Evgeny 6db79808aa smp server: use COPY to import store log to postgres db, improve concurrency and error handling (#1487)
* smp server: use COPY to import store log to postgres db

* compact queues when importing to postgres

* mempty

* version

* handle errors while expiring, mask async exceptions while getting queue

* whitespace

* version
2025-03-18 09:40:22 +00:00
Evgeny f4b55bfc0c smp server: CLI option to compact store log with PostgreSQL database (#1484)
* smp server: CLI option to compact store log with PostgreSQL database

* version

* fix test
2025-03-16 18:47:02 +00:00
Evgeny fe64d42db1 smp server: split postgres support to a separate executable, to not require postgres library in the main binary (#1482)
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary

* comments

* enable server_postgres flag by default, add CPP option to test

* refactor

* change default for server_postgres to False

* diff
2025-03-16 11:37:25 +00:00
Evgeny 1b5a9f3b0c smp server: do not cache all queues from database while processing expirations (#1483)
* smp server: expire only active queues

* version

* do not cache all queues while processing expirations

* refactor

* foldWithOptions_

* version

* use shared lock when expiring all queues

* use TMVar

* comment

* rename

* remove fold options

* do not create locks in the Map for temporarily loaded queues

* fix

* revert version
2025-03-15 20:11:30 +00:00
Evgeny fdf8bd7ee2 smp server: mask database store operations from async exception for state consistency (#1481)
* smp server: mask database store operations from async exception for state consistency

* remove unused, comments
2025-03-13 13:57:46 +00:00
Evgeny 019a32a623 smp server: optionally append store log with postgres storage (without loading and compacting, for debugging during migration) (#1480)
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)

* refactor

* remove comment
2025-03-13 10:25:56 +00:00
Evgeny d44f09d111 tests: pass AStoreType to agent test as a parameter (#1479)
* tests: run agent tests with PostgreSQL SMP servers

* agent tests with postgres database

* enable tests

* fix store log tests

* fix test
2025-03-12 21:47:44 +00:00
Evgeny 6b5de2c51b smp server: exception on invalid migration, check queue storage settings on server start (#1478)
* smp server: check queue storage settings on server start

* fix incorrect postgres migration not throwing exception
2025-03-10 17:28:26 +00:00
shum 9410fb6f16 docker: add libpq dep 2025-03-10 10:03:44 +00:00
Evgeny 7b42aaa132 smp server: expire messages in postgres database, mark queues as deleted, combine tables (#1471)
* smp server: expire messages in postgres database

* tty

* fail if nothing updated in db

* remove old deleted queues

* index

* fix tests
2025-03-10 09:31:50 +00:00
Evgeny Poberezkin e4b9aa9746 Merge branch 'master' into postgres 2025-03-09 21:58:39 +00:00
Evgeny Poberezkin aa26a55df4 Merge branch 'master' into stable 2025-03-09 21:53:03 +00:00
sh 6e505f5c0b scripts: add script to reproduce binaries locally (#1477) 2025-03-08 20:51:20 +00:00
Evgeny Poberezkin a491a1d878 6.3.0.8 2025-03-07 14:30:00 +00:00
shandEvgeny Poberezkin 36f5539b9a ci: introduce reproducible builds (#1476)
* ci: introduce reproducible builds

* ci: return 20.04

* smp server: increase timing test threshold

* ci: test outside docker

* ci: fix test step

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-03-07 14:15:18 +00:00
Evgeny 1a2afe8bfd agent: fix JSON encoding for protocol errors to be compatible with iOS (#1475) 2025-03-06 08:02:27 +00:00
Evgeny Poberezkin 9fece9ce3d 6.3.0.7 2025-03-02 22:45:07 +00:00
Evgeny Poberezkin 2e2ede5968 Merge branch 'master' into postgres 2025-03-02 22:35:28 +00:00
Evgeny 205d4ead1c smp server: remove store log backups when server starts (#1472) 2025-02-27 07:39:05 +00:00
Evgenyandspaced4ndy 80a070a8ea smp server: update STM cache only after PostgreSQL update (#1470)
* refactor journal store

* update postgres store methods to update STM cache only after DB update

* todos

* drop

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* fix GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-26 11:43:08 +00:00
Evgeny 172540984c agent, smp server: PostgreSQL connection pool (#1468)
* agent, smp server: PostgreSQL connection pool

* simplify, create all connections before start

* rename

* remove imports
2025-02-25 15:33:54 +00:00
Evgenyandspaced4ndy 4dc40bd795 smp server: PostgreSQL queue store (#1448)
* smp server: queue store typeclass

* parameterize JournalMsgStore

* typeclass for queue store

* postgres WIP

* compiles, passes tests

* remove StoreType

* split migrations

* progress

* addQueueRec

* reduce type spaghetti

* remove addQueue from typeclass definition

* getQueue

* test postgres storage in SMP server

* fix schema

* comment

* import queues to postgresql

* import queues to postgresql

* log

* fix test

* counts

* ci: test smp server with postgres backend (#1463)

* ci: test smp server with postgres backend

* postgres service

* attempt

* attempt

* empty

* empty

* PGHOST attempt

* PGHOST + softlink attempt

* only softlink attempt

* working attempt (PGHOST)

* remove env var

* empty

* do not start server without DB schema, do not import when schema exists

* export database

* enable all tests, disable two tests

* option for migration confirmation

* comments

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-24 15:47:27 +00:00
Evgeny Poberezkin f9d7b1eebc 6.3.0.6 2025-02-23 19:42:16 +00:00
Evgeny ffbc733d58 smp server: remove duplicate progress log (#1466) 2025-02-23 19:40:49 +00:00
Evgeny 2286726d72 smp server: start options maintenance and skip-warnings (#1465)
* smp server: start options `maintenance` and `skip-warnings`

* ignore invalid parsing of the last lines

* parsingErr

* fix
2025-02-22 19:26:03 +00:00
spaced4ndy 1b8110a332 xftp server: restore file status from log (#1461)
* xftp server: restore file blocking info from log

* fix parse

* rework

* update

* rename
2025-02-21 22:43:56 +00:00
sh dad7e1b60c simplex-servers-update: download scripts from tag (#1457)
* simplex-servers-update: download scripts from tag

* set remote version when re-executing the update script

* safeguard variables

* additional checks
2025-02-20 09:26:03 +00:00
Evgeny 72c2ddcf57 agent: move migrations (#1459) 2025-02-18 23:39:29 +00:00
Evgeny a75e138965 smp server: remove empty queues journals when expiring messages of idle queues (#1458)
* smp server: remove empty queues journals when expiring messages of idle queues

* remove unnecessary update

* ci: update action

* rename
2025-02-18 20:04:58 +00:00
Evgeny fa319d798a smp server: remove empty journals when opening message queue (#1456)
* smp server: remove empty journals when opening message queue

* update, do not backup state

* test

* version

* do not close queue state when queue is opened for writing

* comment

* quota = 4

* refactor openMsgQueue to prevent extra state backups

* use interval in config

* version, expire backups after 5 min

* refactor

* test
2025-02-17 23:11:34 +00:00
Evgeny Poberezkin c192339af9 6.3.0.5 2025-02-15 15:36:44 +00:00
Evgeny fa67d128d1 agent: fix deleting messages after delivery to avoid deleting shared message bodies (#1455)
* agent: fix deleting messages after delivery to avoid deleting shared message bodies

* fix, comments

* rename

* comment
2025-02-15 12:40:10 +00:00
Evgeny 55ff581655 SMP client: dont block on writing to sending queues (#1454)
* SMP client: dont block on writing to sending queues

* only fork if full
2025-02-15 11:31:05 +00:00
sh 560b257af7 scripts: simplex-servers-update menu to update only server binaries etc. (#1445)
* scripts/simplex-servers-update: refactor

* simplex-servers-update: don't check empty variable

* simplex-servers-update: minor decoration change

* simplex-servers-update: add new line when reexecuting script

* simplex-servers-update: refactor menus

* simplex-servers-update: fix re-executing

* simplex-servers-update: fix menus

* simplex-servers-update: do not rerended menu every time

* simplex-servers-update: update menus logic

* simplex-servers-update: ask permission to update only if versions differ

* simplex-servers-update: skip empty variable

* simplex-servers-update: include while loop

* simplex-servers-update: update re-executing skip logic

* simplex-servers-update: ignore empty

* simplex-servers-update: s/while/if

* simplex-servers-update: ignore empty variable

* simplex-servers-update: update skipping scripts

* simplex-servers-update: minor fixes

* simplex-servers-update: add spaces

* simplex-servers-update: safeguard files and add cleanup logic

* simplex-servers-update: update menus

* simplex-servers-update: fix binary check

* simplex-servers-update: adjust erasing lines

* simplex-servers-update: simplify bin update logic

* simplex-servers-update: fix binary path

* simplex-servers-update: format errors

* scripts/main/simplex-servers-update: initial refactor2

* simplex-servers-update: incr update

* simplex-servers-update: initial dynamic menu

* simplex-servers-update: misc fixes

* simplex-servers-update: fix VER

* simplex-servers-update: add VER check

* simplex-servers-update: abort early if there's no options available

* simplex-servers-update: add new helper function and update_misc

* simplex-servers-update: pass selection

* simplex-servers-update: non-interactive usage

* simplex-servers-update: add help menu

* simplex-servers-update: misc fixes and new binary update logic

* simplex-servers-update: local path check

* simplex-servers-update: fix change check

* simplex-servers-update: OLF -> OLD

* simplex-servers-update: always return code from functions

* simplex-servers-update: new menu flow

* simplex-servers-update: fixes

* simplex-servers-update: download function + fixes

* simplex-servers-update: s/bins/binaries

* simplex-servers-update: download message

* simplex-servers-update: cleanup
2025-02-15 10:31:19 +00:00
spaced4ndyandEvgeny Poberezkin 7ac80bffcb agent: store shared message body only once (when it is the same across messages when batching) (#1453)
* agent: store shared message body only once (when it is the same across messages when batching)

* rename

* refactor

* refactor

* save bodies and messages in single transaction

* comment

* comment

* comment

* box

* mapME

* box

* ValueOrRef

* remove instances

* refactor

* comments

* test

* refactor

* mapAccumLM compatibility with ghc 8.10.7

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 18:01:40 +00:00
spaced4ndyandEvgeny Poberezkin 0d8a1a2879 agent: encrypt messages on delivery (#1446)
* agent: save message body once (plan, schema)

* split

* new type

* bs

* encrypt on delivery

* schema

* fix test

* check pad size

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-14 12:35:18 +00:00
Evgeny Poberezkin bd97cb0449 6.3.0.4 2025-02-10 15:12:54 +00:00
Evgeny a58d3540ad agent: refactor migrations (#1452)
* agent: refactor migrations

* fix postgres
2025-02-07 17:17:53 +00:00
Evgenyandspaced4ndy 5dbe6337ea ntf server: additional statistics, new invalid token reasons (#1451)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* ntf server: additional statistics

* swap

* version

* more stats

* test, verify invalid

* rename

* exclude test token from stats

* increase delay

* handle invalid token in retry, more reasons

* focus tests

* disable new tests in CI

* fix

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-02-07 12:19:11 +00:00
spaced4ndyandEvgeny Poberezkin b633f89c1a agent: check ntf token status on registration (#1450)
* agent: check ntf token status on registration

* remove check

* update on check

* refactor

* version

* fix

* test, verify invalid

* rename

* increase delay

* disable new tests in CI

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-02-07 11:36:29 +00:00
Evgeny 944a22a2fb ntf server: record token invalidation reason, add date of the last token activity (#1449)
* ntf server: record token invalidation reason, add date of the last token activity

* update time

* rename

* optional

* include token ID in delivery error

* version

* protocol version

* fix, log error
2025-02-06 17:11:35 +00:00
Evgeny ce24f83b64 refactor STM queues (#1447) 2025-02-05 12:04:27 +00:00
Evgeny Poberezkin 45373e7f1f 6.3.0.3 2025-01-31 13:01:03 +00:00
sh efe71cd598 docker: refactor (#1438)
* docker: refactor

* github/docker: bump actions and adjust smp ports
2025-01-30 09:22:13 +00:00
Evgeny ae41717b9b smp server: use origin client version when processing proxied command, fixes old client sending to new server (#1443)
* smp server: pass origin client version when processing proxied command, fixes old client sending to new server

* version

* version
2025-01-28 22:04:46 +00:00
Evgeny e78ab60c97 build: fix postgres (#1444) 2025-01-28 22:02:41 +00:00
Evgeny b3c8358a43 agent: combine connection deletion events (#1442) 2025-01-25 08:52:40 +00:00
sh be81fe1f74 scripts/systemd: update services (#1440) 2025-01-24 13:37:56 +00:00
sh 817f5e1737 scripts/systemd: update services (#1440) 2025-01-24 13:35:33 +00:00
Evgeny eda9e36c82 agent: track queries (#1439) 2025-01-24 10:31:50 +00:00
Evgeny Poberezkin 2318975375 6.3.0.2 2025-01-20 16:38:14 +00:00
Evgeny 298666380f servers: handle accept error ECONNABORTED (#1437) 2025-01-20 16:32:42 +00:00
Evgeny 6a9075141f xftp server: use recipient ID in control port to delete and block files, smp server: fix version negotiation (#1434)
* xftp server: use recipient ID in control port to delete and block files

* cap smp proxy agent version at 10

* version

* fix prometheus

* fix

* remove old version support

* log connection parameter on error

* tests

* log sent command tag

* log error and client version

* cap proxy version for previous destination server

* comment, test

* remove logging tag

* remove logs

* version

* SMP version 14

* version

* remove comments

* version
2025-01-20 13:45:49 +00:00
spaced4ndy 488c7082f3 agent: store interface (#1436) 2025-01-20 17:02:39 +04:00
spaced4ndy fdde9863cd agent: add reopenStore function for postgres; run notification tests with postgres (#1435) 2025-01-17 16:27:37 +04:00
Evgeny Poberezkin 9404a3ab63 xftp server: block stats 2025-01-14 10:46:27 +00:00
Evgeny Poberezkin dadf6ec5b6 6.3.0.1 2025-01-13 08:39:41 +00:00
Evgeny c528ea4f31 smp server: update versions (#1433) 2025-01-12 22:11:18 +00:00
Evgeny 3d4e0b06c0 servers: blocking records for content moderation (#1430)
* servers: blocking records for content moderation

* update

* encode BLOCKED as AUTH in old versions

* update

* unblock queue command

* test, status command
2025-01-12 19:34:00 +00:00
spaced4ndy 9d9ec8cd0b agent: remove unused functions (#1432)
* agent: remove unused functions

* cleanup imports
2025-01-10 07:59:17 +00:00
Evgeny 992b42e922 agent: option to enable/disable vacuum after SQLite migration (#1429) 2024-12-28 21:06:42 +00:00
Evgeny 3cf9dacbc0 Merge pull request #1428 from simplex-chat/ep/postgres-compat
agent: restore methods for backwards compatibility with simplex-chat
2024-12-28 11:39:18 +00:00
Evgeny Poberezkin cfde5932fd agent: restore methods for backwards compatibility with simplex-chat 2024-12-28 10:33:22 +00:00
Evgeny 69fb9a9e0e Merge pull request #1417 from simplex-chat/postgres
agent: support postgres backend
2024-12-27 19:52:18 +00:00
Evgeny Poberezkin 0a118f5631 Merge branch 'master' into postgres 2024-12-27 11:58:19 +00:00
spaced4ndy 0b986c1158 postgres: export fromOnlyBI 2024-12-26 20:13:27 +04:00
Evgeny Poberezkin 184a95cd2a 6.3.0.0 2024-12-26 13:54:38 +00:00
spaced4ndy b7a8df40c4 Merge branch 'master' into postgres 2024-12-25 12:15:41 +04:00
spaced4ndyandEvgeny Poberezkin bf28902327 agent: reuse ratchet on repeat join (#1426)
* agent: reuse ratchet on repeat join

* check status

* update

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-25 00:07:45 +00:00
Evgeny Poberezkin 426bf68763 6.2.2.0 2024-12-24 21:04:58 +00:00
spaced4ndy e557dc86e7 Merge branch 'master' into postgres 2024-12-20 17:18:54 +04:00
Evgeny Poberezkin f5cef2556b build: move type to client_library module 2024-12-20 13:12:11 +00:00
spaced4ndyandEvgeny Poberezkin cf66aadc20 postgres: store implementation, conditional compilation (#1421)
* postgres: implementation wip

* to from field

* agent store compiles

* methods

* create store

* tests wip

* migration tests pass

* tests compile

* fix tests

* tests wip

* bool int

* tests wip

* tests wip

* more boolint

* more fixes

* more fields pass

* more fixes

* binary

* instances, binary

* test passes

* remove todos, more tests pass

* fix conflict

* fix bool

* fix sequence breaking

* fix insertedRowId

* skip ratchet re-synchronization tests

* after test

* file tests

* after test

* rename

* remove comment

* format

* remove unused

* suppress notices

* fixes

* move

* fix

* instance

* instance2

* fix

* instances

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-12-20 15:54:58 +04:00
sh 77a5ed2ec6 6.2.1.0 (#1424) 2024-12-19 20:45:17 +00:00
Evgenyandsh 4d640c16aa smp server: log prometheus metrics (#1411)
* smp server: log prometheus metrics

* save metrics

* diff

* lines

* version

* do not include Prometheus into client

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* corrections

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

* add timestamp to metrics

* remove type

* remove version

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2024-12-18 11:19:13 +00:00
Evgeny 0a827307d6 build: remove some modules from the client build (#1418)
* build: remove some modules from the client build

* remove websockets from client_library
2024-12-14 15:22:06 +00:00
spaced4ndy 64149599de postgres: db interfaces wip (sqlite passes) (#1419) 2024-12-12 17:42:58 +04:00
spaced4ndy 07be469f49 postgres: schema (#1416) 2024-12-10 18:53:01 +04:00
Evgeny Poberezkin 79e9447b73 6.2.0.7 2024-12-07 14:36:50 +00:00
spaced4ndy 9893935e7c agent: fix acceptContact inconsistent state after app restart (#1412) 2024-12-05 19:27:21 +04:00
Evgeny Poberezkin 966b9990e0 6.2.0.6 2024-12-04 19:57:11 +00:00
spaced4ndy bef1e38295 xftp: add debug info to "too many recipients" error (#1410) 2024-12-04 19:56:33 +00:00
Evgeny Poberezkin 4b43cb8054 6.2.0.5 2024-12-03 12:33:32 +00:00
Evgeny 38ad3c046e agent: function to check if contact address is the same (#1409) 2024-12-02 11:41:06 +00:00
Evgeny b2afe6f40c rfc: moving queue records to journal storage and queue blobs, expiring messages in journal storage (#1407)
* rfc: moving queue records to journal storage and queue blobs

* rfc: expiring messages in journal storage

* add comment
2024-11-28 12:22:52 +00:00
Evgeny Poberezkin 601620bdde 6.2.0.4 2024-11-22 19:50:04 +00:00
Evgeny 97104988a3 server: support server roles and operators (#1343)
* server: support server roles and operators

* make server operator optional

* allRoles

* fix test

* different server host in tests

* remove ServerCfg fields used only in UI

* comments

* choose different server for invitation when connecting via address

* fix test in ghc8107

* simplify
2024-11-22 16:43:37 +00:00
Evgeny 45333bd340 smp server: do not open/read journal message queues that are known to be empty (#1406)
* smp server: do not open/read journal message queues that are known to be empty

* cleanup

* version

* close empty queues on first subscription

* revert version
2024-11-22 12:29:31 +00:00
Evgeny bbcb1abfda agent: treat blocked STM and other critical errors that offer restart as temporary (#1405) 2024-11-21 22:01:19 +00:00
Evgeny Poberezkin a64c1aa2c4 6.2.0.3 2024-11-17 09:45:17 +00:00
Evgeny 21fbbf9106 smp server: update message counts during message expiration, increase idle interval (#1404)
* smp server: update message counts during message expiration, increase idle interval

* version

* fix

* flip results

* version
2024-11-17 09:44:40 +00:00
Evgeny 17a0be10fa smp server: expire messages in idle message queues (including not opened) (#1403)
* smp server: expire messages in idle message queues (including not opened)

* use message expiration interval

* simpler

* version

* remove version
2024-11-15 10:26:24 +00:00
Evgeny Poberezkin 3017d14392 6.2.0.2 2024-11-07 08:13:45 +00:00
Evgeny d3275cef48 smp server: combine messages and queue storage to optimise performance, prevent race condition when deleting queue and to avoid "orphan" messages. (#1395)
* smp server: combine queue and message store into one class (WIP)

* keep deleted queue tombstones to prevent race conditions and errors when restoring

* move store log from server to store implementations

* STMQueueStore type class

* fix store closed when messages expired, handle store writing errors

* types

* version

* fix recovery from missing write journal, tests

* version
2024-11-07 08:09:11 +00:00
Evgeny Poberezkin 0fd4aa1e23 6.2.0.1 2024-11-03 21:10:45 +00:00
Evgeny 7c21945569 smp server: option to skip expiring messages on start, read queue state file end only (#1400) 2024-11-03 21:08:07 +00:00
ffecf200d4 build: client_library flag to build only used modules, remove package yaml (#1394)
* move dependencies to library

* remove package.yaml

* remove auto-generated comment

* cleanup target definitions

* add executable options

* add test rtsopts

* revert common options

* don't hide unused packages warning

* Add client_only cabal flag (#1396)

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-11-02 15:34:33 +00:00
Evgeny 58d3c160ef smp server: fix logging progress when importing journal (#1393) 2024-10-28 14:22:18 +00:00
Evgeny Poberezkin 326cf08efa 6.2.0.0 2024-10-28 10:28:54 +00:00
Evgeny Poberezkin d49dc201c3 smp server: log file name on hClose error 2024-10-28 10:26:24 +00:00
Evgeny 6ff4c3f1ce Merge pull request #1375 from simplex-chat/journal-store
smp server: journal message store
2024-10-26 17:03:19 +01:00
Evgeny 5940514f40 smp server: remove queue from map when closing, test (#1392)
* smp server: remove queue from map when closing, test

* remove print

* refactor
2024-10-26 16:14:55 +01:00
Evgeny Poberezkin 916150742e Merge branch 'master' into journal-store 2024-10-26 12:58:32 +01:00
Evgeny Poberezkin 1735b619e8 6.1.3.0 2024-10-26 12:57:43 +01:00
Evgeny 6b61d101dc smp server: fix restoring notifications (#1391) 2024-10-26 12:56:29 +01:00
Evgeny 56ca7bf30e smp server: do not log progress when server starts (#1390)
* smp server: do not log progress when server starts

* test
2024-10-26 10:33:18 +01:00
Evgeny Poberezkin 2a218675b2 Merge branch 'master' into journal-store 2024-10-26 08:12:07 +01:00
Evgeny 8b9eb49b57 smp server: log operation of restore error (#1389)
* smp server: log operation of restore error

* space

* rename

* revert version
2024-10-26 08:11:07 +01:00
Evgeny 3860341584 rfc: iOS notifications 2 (#1334) 2024-10-25 21:42:13 +04:00
Evgeny Poberezkin 9078118a6d Merge branch 'master' into journal-store 2024-10-25 15:43:04 +01:00
Evgeny Poberezkin a7f5297ad1 6.1.2.0 2024-10-25 15:41:30 +01:00
Evgeny 870a60fc5d smp server: reduce memory usage when starting, all servers: prevent losing state on termination while compacting state (#1385)
* smp server: reduce memory usage when starting

* servers: prevent losing state when compacting is incomplete (terminated abnormally)

* fix test

* fix/refactor

* message

* remove .bak

* simpler
2024-10-25 15:39:59 +01:00
spaced4ndyandEvgeny Poberezkin a8471eed5b ntf: get messages for multiple last notifications (#1378)
* ntf: get messages for multiple last notifications (#1352)

* ntf: separate get ntf conns api (#1379)

* ntf: separate get ntf conns api

* nonempty

* update

* update

* remove single get api

* fix test

* refactor

* refactor

* ntf: batch get connections (#1387)

* ntf: batch get apis

* works

* fix

* fix

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-25 15:39:41 +04:00
Evgeny a3351ad060 smp server: fix export of journal storage (#1384)
* smp: fix export of journal storage

* catch errors when exporting messages
2024-10-25 09:11:47 +01:00
Evgeny 2322f5bf59 smp server: logging format, mask/handle exceptions during journal store operations (#1381)
* smp server: fixed logging format for journal store errors

* version

* colon

* logs

* refactor

* space

* remove comment

* log file name in fixFileSize

* logError

* stricter

* process all queues more efficiently

* use monoid for queue processing

* expire messages concurrently

* concurrently 2

* Revert "concurrently 2"

This reverts commit c1aee1f22c.

* Revert "expire messages concurrently"

This reverts commit fc53137cdb.

* show queue directory or ID in errors

* foldM

* mask_

* try

* mask more

* refactor

* command to delete journal

* uninterruptibleMask_ when writing to state file

* fix ghc8.10.7

* version

* revert version
2024-10-24 19:44:47 +01:00
Evgeny c9c075fd49 smp server: merge quota messages and set queue to "over quota" state after restoring, server tests with journal and memory store (#1380)
* smp: run server tests with journal and memory store

* merge quota messages, set queue to "over quota" state after restoring

* fix test
2024-10-23 09:17:23 +01:00
Evgeny 1484523164 smp server, journal store: check file size on append, validate queue state (#1377)
* smp server: journal store queue state validation

* validate state on update, refactor

* handle message parsing errors

* typo

* fix test, throw exception when file is smaller than expected

* fix

* fix test in ghc 8.10.7
2024-10-22 12:09:51 +01:00
Evgeny dc17402896 smp server: updated journal store API (#1374)
* core: updated journal store API

* parameterize JournalState by read/write type

* collect stats when importing/expiring

* compare stored stats and store for messages and notifications

* simplify
2024-10-21 16:50:07 +01:00
Evgeny 0414ea59f0 smp server: journal message store (#1370)
* smp server: remove STM function from MsgStore

* polymorphic MsgStore

* jourmal storage for messages (WIP)

* more journal, test setup

* writeMsg

* test

* tryDelMsg

* delMsgQueue

* remove MsgStoreClass instance of existential wrapper for Msg stores

* store config

* extract common logic out of store instances

* add store type to config

* open journals, cache last message, tests pass

* CLI commands

* refactor import/export messages

* cli commands to import/export journal message store

* export journal without draining, import/export tests

* journal command

* import/export progress

* better progress info

* only log queue state once when importing

* logs

* handle IO errors in journal store, return as STORE error

* recover from state file errors

* fix message files after crash

* fix messages folder
2024-10-21 11:50:30 +01:00
Evgeny Poberezkin 967afaf802 6.1.1.0 2024-10-18 12:47:27 +01:00
Evgeny 77ee9bea38 smp protocol: add STORE error (#1372) 2024-10-18 10:13:01 +01:00
Evgeny e0cd28a65a smp server: stop server faster (#1371)
* smp server: stop server faster

* .702

* asks

* fix leak

* kill client when server stopped

* remove version

* fix version
2024-10-18 09:12:59 +01:00
Evgeny Poberezkin 7b6c86c6c1 docs: correction 2024-10-14 08:32:19 +01:00
Evgeny Poberezkin 1d0a418011 rfc: rename file 2024-10-12 17:12:45 +01:00
Evgeny Poberezkin 9253f5647a Merge branch 'stable' 2024-10-12 16:30:54 +01:00
Evgeny Poberezkin c41bfe831d 6.1.0.7: changelog for v6.1.0 2024-10-11 13:01:12 +01:00
Alexander BondarenkoandEvgeny Poberezkin c9af7e327e smp server: graceful shutdown on SIGINT (#1360)
* smp-server: graceful shutdown on SIGINT

* .501

* clean up

* remove version

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-11 12:49:15 +01:00
Evgeny 8870442830 smp server: reject SKEY with different key earlier (at verification step), add tests (#1366) 2024-10-11 12:13:52 +01:00
spaced4ndyandEvgeny Poberezkin 9ad26c97a1 agent: prepareConnectionToAccept api (#1365)
* agent: prepare conn (plan)

* implement

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-11 11:05:14 +01:00
spaced4ndy e1201f0967 agent: test db indexes (#1364) 2024-10-09 15:16:08 +04:00
Evgeny Poberezkin ca2202e3c8 6.1.0.6 2024-10-07 22:41:53 +01:00
Evgeny f24ee70138 clients: do not send SNI (#1363) 2024-10-07 22:40:30 +01:00
Evgeny Poberezkin 8981cae3c2 ci: always generate change log 2024-10-07 22:22:02 +01:00
Evgeny Poberezkin b8971a31bc 6.1.0.5 2024-10-07 10:24:19 +01:00
Evgeny e01398a31a agent: send MSGNTF on NO_MSG error (#1362)
* agent: send MSGNTF on NO_MSG error

* simplify
2024-10-07 10:22:58 +01:00
Evgeny f871f20172 smp server: fix notification delivery (#1350)
* .401

* stats for undelivered notifications

* logs, stats

* control port show ntf client IDs

* check that Ntf client is still current and that queue is not full, drop notifications otherwise

* prevent losing notifications when client is not current or queue full

* add log when no notifications, remove some logs

* reduce STM transaction

* revert version change
2024-10-07 09:01:28 +01:00
Evgeny 80d3518d55 tests: store log (#1348)
* 6.1.0.4

* tests: store log

* random

* more tests
2024-10-06 21:21:08 +01:00
spaced4ndyandEvgeny Poberezkin 49935473ae ntf: agent to notfiy MSGNTF on successful ack (#1355)
* ntf: agent to notfiy MSGNTF on successful ack

* move msgntf

* release lock earlier

* fix tests

* do not release get lock

* fix

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-10-06 19:48:22 +01:00
Evgeny 0705d05078 smp: transport block encryption (#1317) 2024-10-06 18:05:06 +01:00
Evgeny 97d45fb2a6 smp server: pass server information via CLI during server initialization (#1356)
* smp server: pass server information via CLI during server initialization

* more info

* enable client expiration by default, disable port 8000

* update
2024-10-06 12:34:44 +01:00
Evgeny 50e4abf836 ntf server: control port (#1354)
* ntf server: control port

* version .405

* control

* use own_server_domains from INI file

* fix subs by server in control port

* bigger queues

* ntf server: only print subscriptions per own server when they are > 0

* fix tests

* revert version change

* dont import listThreads in ghc 8.10.7
2024-10-06 12:33:14 +01:00
spaced4ndy b130ea439a server: fix msgGetNoMsg stat (#1358) 2024-10-04 17:19:05 +04:00
sh 0e921d3258 scripts: preparations for new deployments (#1357)
* scripts/install: allow specifying version

* scripts/update: allow specifying version

* scripts/services: allow 443 for smp, allow infinite startup
2024-10-04 09:57:07 +01:00
Evgeny d0f65329fb ntf server: enable pings on ntf subscriptions (#1353) 2024-10-03 12:12:55 +01:00
Evgeny Poberezkin 61b2b9df1a Re-commit: smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
This reverts commit 0ba3e69872.
2024-10-02 12:23:46 +01:00
Evgeny Poberezkin 04e4a37d85 6.1.0.4 2024-10-02 09:51:29 +01:00
Evgeny Poberezkin 0ba3e69872 Revert "smp server: persist notifications to avoid losing them when ntf server is offline (#1336)"
This reverts commit 0739f7b702.
2024-10-02 09:49:43 +01:00
Evgeny Poberezkin 7488923fc3 6.0.6.0 2024-10-02 09:24:29 +01:00
Evgeny cf559ee43f smp server: fix notifiers lost when compacting store log (#1347) 2024-10-02 09:24:04 +01:00
Evgeny 2ab0c2a7c6 smp server: fix notifiers lost when compacting store log (#1347) 2024-10-01 22:28:01 +01:00
spaced4ndy 243548631a ntf: repeat failed token deletions (#1346) 2024-10-01 16:30:14 +04:00
Evgeny Poberezkin da79d544cf 6.1.0.3 2024-09-30 18:27:14 +01:00
spaced4ndy e808825c95 ntf: fix endless worker loop (#1345) 2024-09-30 15:29:14 +01:00
Evgeny Poberezkin a9576935cf 6.1.0.2 2024-09-29 18:54:04 +01:00
spaced4ndyandEvgeny Poberezkin 9cf6c97137 ntf: batch ntf worker command processing (#1337)
* ntf: batch ntf worker command processing

* remove comment

* change batch size

* wip

* catch

* refactor

* refactor

* batch check

* refactor

* reschedule

* increase ntfSubCheckInterval

* first check interval

* check more statuses

* refactor, remove foldr'

* refactor 2

* refactor client

* refactor 3

* ntf server: improve support for batched commands (#1340)

* re-create ntf subscriptions on NTF AUTH errors

* name

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-29 18:50:38 +01:00
Evgeny 0739f7b702 smp server: persist notifications to avoid losing them when ntf server is offline (#1336)
* ntf server: types for storing notifications

* fix tests

* remove comments

* batch NMSGs test

* fix test

* thread to expire notifications

* persist notifications on restart

* optimize

* refactor

* remove reverse

* control port stats
2024-09-29 18:50:16 +01:00
Evgeny e12710fa55 smp server: show version on server page (#1341) 2024-09-29 18:46:38 +01:00
EvgenyandIC Rainbow 2a120dfe57 smp server: Allow serving HTTPS and transport on the same port (v2) (#1327)
* smp-server: Allow serving HTTPS and transport on the same port

* update rfc

* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started

* diff

* combine HTTP and SMP on the shared port

* Update to SockAddr

* Fix params and web.https parser

* Switch fork urls

* WIP: add smpServerTestStatic test

* Update warp-tls repo

* shared connection tests

* cleanup

* Add protocol tests

* rename cert file, enable both ports and web by default

* terminate with message on missing credentials

* test cert file

* client option to use port 443 as default SMP port

* use SNI in non-SMP clients

* supported

* remove TODO

* advice

* fix test build

* Add RSA-4096 check for web creds, fix test

* Remove directory listing from static app

* message

* messages

* update log tests

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-09-28 23:15:17 +01:00
Evgeny 3c18c4b66a servers: refactor TLS credentials (#1326)
* servers: refactor TLS credentials

* provide server credentials in SNI hook

* determine TLS server params dynamically, when starting the server

* remove alpn from TransportServerConfig to decide it dynamically where server is started
2024-09-28 22:21:08 +01:00
Evgeny 21eee2b548 smp servers: separately account for subscribed clients, to disconnect those without subscriptions more effectively (and to be able to deliver notifications to subscribed ntf servers) (#1339) 2024-09-28 19:59:53 +01:00
Evgeny Poberezkin 03168b9fbf 6.1.0.1 2024-09-26 12:23:31 +01:00
spaced4ndy 41e1b2bd39 ntf: use common function for differentiating temporary errors (#1333) 2024-09-25 18:40:59 +01:00
Evgeny 1afa0206a5 smp: increase server and default agent queue sizes (#1335) 2024-09-25 18:33:45 +01:00
spaced4ndyandEvgeny Poberezkin 81fcdf8ac9 ntf: batch smp worker command processing (#1331)
* ntf: refactor smp worker (separate command processing into steps)

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-25 15:00:06 +04:00
Evgeny 1f67b403de agent: function to get multiple work items (#1330) 2024-09-25 10:55:45 +01:00
Evgeny 8e7f3f7b27 agent: return full argument with batched results (#1332) 2024-09-24 16:16:52 +01:00
Evgeny Poberezkin e6e9f1d5a5 Merge branch 'stable' 2024-09-24 12:36:05 +01:00
Evgeny Poberezkin 4268b90763 6.0.5.0 2024-09-24 12:34:46 +01:00
Evgeny 856137b763 rfс: efficient message storage (#1295)
* rfs: efficient message storage

* typos

* update
2024-09-24 12:29:44 +01:00
Evgeny d0ee505c3a rfc: protocol for SMP queue data blobs and short links (#1309)
* rfc: protocol for SMP queue data blobs and short links

* update

* private rendezvous
2024-09-24 12:29:11 +01:00
Evgeny 7dcac19a67 xrcp: session encryption with forward secrecy (#1328)
* xrcp: session encryption with forward secrecy

* update doc

* accept keys as parameters

* docs: counter

* spec corrections
2024-09-24 09:22:26 +01:00
Evgeny 22260cd719 agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes (#1321)
* agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes

* fix test
2024-09-22 22:22:05 +01:00
spaced4ndyandEvgeny Poberezkin bef11e4cbe agent: ntf errs (#1316)
* agent: ntf errs

* refactor

* errs

* refactor

* simplify

* inline

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-20 20:08:42 +04:00
Evgeny Poberezkin d5a65656f5 test: socks proxy parameter encoding 2024-09-16 13:15:30 +01:00
Evgeny Poberezkin 309ef3766c 6.1.0.0 2024-09-16 07:28:50 +01:00
Evgeny Poberezkin bde7b97f6a Merge branch 'stable' 2024-09-15 21:38:22 +01:00
Evgeny 398dbeedea servers: allow multiple server ports for SMP and Ntf servers (#1319)
* servers: allow multiple server ports for SMP and Ntf servers

* fix
2024-09-15 21:37:09 +01:00
Evgeny fa772af6c6 agent: support socks proxy without isolate-by-auth, with and without credentials (#1320)
* agent: support socks proxy without isolate-by-auth, with and without credentials

* add unit tests

* make xftp use correct SOCKS credentials

* rename

* support ipv6 in brackets, test parsing

* constant

* textToHostMode

* space
2024-09-15 21:36:31 +01:00
Evgeny Poberezkin 9ebdd2b0ea Merge branch 'stable' 2024-09-14 19:00:36 +01:00
Evgeny bec4e5e038 smp: reduce max message sizes (#1318) 2024-09-14 17:34:29 +01:00
spaced4ndy ea67b34ba7 ntf: batch supervisor commands (#1313) 2024-09-13 15:26:57 +04:00
spaced4ndyandEvgeny Poberezkin e247f691cb agent: do not reuse notifier credentials when re-creating notification subscription; on deleting ntf sub delete directly on smp server skipping ntf server (#1311)
* agent: change notifier credentials when re-creating notification subscription

* skip ndel

* enable tests

* fix race

* Revert "fix race"

This reverts commit ed9b18e8a7.

* delete record

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-09-12 14:02:51 +04:00
Evgeny Poberezkin 896b2425a4 xrcp: use SHA3-256 in hybrid key agreement (#1302)
This reverts commit 62133ceb24.
2024-09-11 19:32:40 +01:00
spaced4ndy 30e15b881f agent: don't prohibit deletion of notifications token if different is passed (fixes turning notifications off after importing database) (#1308)
* agent: don't prohibit deletion of notifications token if different is passed (fixes turning notifications off after importing database)

* ignore error

* notify
2024-09-11 19:30:59 +01:00
Evgeny Poberezkin 24ded9e5a0 Merge branch 'master' into stable 2024-09-11 19:03:29 +01:00
Evgeny Poberezkin f5e666ae4f 6.0.4.0 2024-09-11 18:51:26 +01:00
Evgeny Poberezkin 62133ceb24 Revert "xrcp: use SHA3-256 in hybrid key agreement (#1302)"
This reverts commit 67d38090ed.
2024-09-11 18:45:44 +01:00
Evgeny 3b50e1fb7d ntf server: only use SOCKS proxy for servers without public address (#1314) 2024-09-11 18:41:40 +01:00
Evgeny 7c25b3b1e0 smp protocol: send DELD when subscribed queue is deleted (#1312)
* smp protocol: send DELD when subscribed queue is deleted

* fix, test

* refactor

* send DELD event only if the client supports it (version 10); send END otherwise

* fix test

* notify on notifier rotation

* increase test delays
2024-09-11 13:16:51 +01:00
Evgeny a70bd02c67 xftp server: round down file creation time to 1 hour (#1310) 2024-09-10 08:14:05 +01:00
Evgeny 990dcec348 smp server: add created/updated/used date to queues to manage expiration (#1306)
* smp server: add created/updated/used date to queues to manage expiration, all: make Map updates strict in value

* remove strict

* remove time precision

* diff

* style

* only update when time changed
2024-09-09 14:53:11 +01:00
Evgeny 75712641ee rfc: stabilize iOS notifications (#1221) 2024-09-09 17:07:32 +04:00
spaced4ndy 946e16339e agent: process last notification from list (#1307) 2024-09-09 16:42:14 +04:00
spaced4ndy 092ed088ca ntf: support for multiple messages encoding (#1305) 2024-09-09 16:03:17 +04:00
sh 7000431249 scripts/servers: update stopscript (#1286)
* scripts/servers: update stopscript

* Major refactoring

* archive -> backups

* simplify logic

* sort files by timestamp
2024-09-09 10:19:17 +01:00
Evgeny dab1980d79 xftp: report receive file error with redirected file ID, when redirect is present (#1304)
* xftp: report receive file error with redirected file ID, when redirect is present

* fix test
2024-09-09 08:08:16 +01:00
Evgeny 344a295845 agent: error when user record is not in database (#1303) 2024-09-08 16:50:22 +01:00
Evgeny 67d38090ed xrcp: use SHA3-256 in hybrid key agreement (#1302) 2024-09-08 15:45:45 +01:00
Evgeny a9e8d02593 server: bind control port server only to 127.0.0.1 for better security (in case of firewall misconfuguration) (#1280) 2024-09-05 13:48:09 +01:00
Evgeny d859f27999 ntf server: remove debug logging (#1284) 2024-09-05 13:26:34 +01:00
Evgeny e86338d555 smp server: fewer map updates on re-subscriptions (#1297)
* smp server: fewer map updates on re-subscriptions

* temp version

* replace Client with ClientId in queues

* version

* version

* comments

* reduce threads when sending ENDs

* revert version
2024-09-05 13:25:41 +01:00
Evgeny 137ebc1cad servers: reduce memory used for period stats (#1298) 2024-09-02 23:12:08 +01:00
Evgeny d84a49b85a smp server: split and reduce STM transactions (#1294) 2024-09-02 17:06:31 +01:00
Evgeny d5efe3406a agent: fix race when sending a message to the deleted connection (#1296) 2024-09-02 15:07:16 +01:00
Evgeny Poberezkin d559a66145 6.0.3.0 2024-08-30 12:55:17 +01:00
Evgeny ce6777b68d newtype for server entity IDs, fix TRcvQueues (#1290)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* newtype for server entity IDs, fix TRcvQueues

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* logServer
2024-08-30 12:50:02 +01:00
Evgeny 655e7ad7d5 smp server: get message queue faster, avoiding STM contention if queue exists, split transaction for notification delivery (#1289)
* put DRG state to IORef, split STM transaction of sending notification (#1288)

* put DRG state to IORef, split STM transaction of sending notification

* remove comment

* remove comment

* add comment

* revert version

* smp server: get message queue faster, avoiding STM contention if queue exists

* IORef for counter

* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"

This reverts commit 517933d189.

* version

* remove IORef

* split notification delivery transations

* revert version
2024-08-30 11:53:22 +01:00
Evgeny 9596a03139 servers: reduce STM transactions (#1287)
* servers: reduce STM transactions

* switch stats and pending ENDs to IORef

* more IORef, split pending ENDs to use in one thread
2024-08-29 13:18:12 +01:00
Evgeny 2e7e476f81 smp server: remove "expensive" stats (#1285) 2024-08-28 18:20:44 +01:00
Evgeny 2c9ad74599 smp server: batch END responses when subscribed client switches (version 2) (#1283)
* fix incorrect entity ID for notification subscription END when queue is deleted

* throttle sending ENDs

* fix stats

* clean up
2024-08-28 10:14:41 +01:00
Evgeny Poberezkin aa60bd6770 use strict Maps, fix stats for sent END batches 2024-08-26 19:35:57 +01:00
Evgeny 16cf5c8628 smp server: stats for END events and for SUB/DEL event batches (#1281)
* smp server: count queued and sent END events

* fix

* shadowing

* stats for batches

* fix
2024-08-26 14:58:18 +01:00
Evgeny Poberezkin 56986f82c8 6.0.2.0 2024-08-24 14:54:32 +01:00
Evgeny 53e829a21c agent: store query errors, reduce slow query threshold to 1ms (#1277)
* agent: collect query errors stats

* simplify

* test

* use microseconds

* parens

* revert change to track all queries, reduce threshold to 1 ms
2024-08-24 14:51:26 +01:00
Evgeny ab4acadcc1 agent: fix stuck connection commands when server is not responding (#1276)
* agent: fix stuck connection commands when server is not responding

* fix

* rename

* test SKEY retry
2024-08-24 13:54:20 +01:00
Evgeny 325e679bc4 ntf server: reduce PING interval to 1 minute, only mark subscriptions with END if it is received by the currently active client (#1275) 2024-08-24 11:34:59 +01:00
Evgeny Poberezkin d81cae712e Merge branch 'master' into stable 2024-08-23 14:41:15 +01:00
Evgeny Poberezkin 1d22608f86 6.0.1.0 2024-08-20 18:46:42 +01:00
Evgeny 39f3cf3082 agent: only create notification subscriptions to connections without errors or with temporary errors (#1270) 2024-08-20 16:05:50 +01:00
sh 5be81c562f scripts: update install/update (#1271) 2024-08-20 15:57:29 +01:00
Evgeny 2bdda1ff1f servers: safe compacting of store logs (#1269)
* servers: safe compacting of store logs

* fix
2024-08-20 13:36:37 +01:00
Evgeny ac930dff30 smp server: possible race when creating client that might lead to memory leak (#1260) 2024-08-20 12:25:58 +01:00
DiogoandEvgeny Poberezkin 1cbf8c0015 agent: support changing user of the new connection (#1267)
* agent: support new connection user id update

* another way for assertion

* add more tests to setConnUserId

* remove fdescribes

* allow rcv connection to change user id

* add functional test to api

* remove fdescribe

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-08-20 10:57:09 +01:00
Evgeny 571d148bdf agent: do not start delivery workers when there are no messages to deliver (#1263)
* agent: use weak ThreadId and forkIO in workers instead of async (reduce memory)

* agent: do not start and exit delivery workers when there are no messages to deliver (#1264)

* agent: exit delivery workers when no messages to deliver

* only start delivery workers when there are pending messages

* fix

* focus test

* enable all tests

* lift

* do not exit workers when there is no work
2024-08-18 21:30:06 +01:00
Evgeny 32a150eb40 agent: enable notifications for all connections (#1262) 2024-08-18 14:07:09 +01:00
EvgenyandAlexander Bondarenko 5ad6e5f2f3 deps: upgrade tls to 1.9 (#1265)
* deps: use tls-2.0

* roll back RCP "cleanup"

* use tls 1.9

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-08-18 13:55:12 +01:00
Evgeny f229e135e3 agent: reduce subscription transactions (#1259)
* agent: reduce subscription transactions

* nub

* remove commented
2024-08-16 10:06:22 +01:00
Evgeny Poberezkin d15d87c111 Merge branch 'master' into stable 2024-08-11 18:06:11 +01:00
Evgeny Poberezkin 0dd52dc69f v6.0.0 (6.0.0.8) 2024-08-09 23:09:09 +01:00
Evgeny 47dd0c4c64 agent: do not send worker crash errors when agent is not active (stopped) (#1256) 2024-08-09 21:10:27 +01:00
Evgeny 7762b4d364 reduce the number of STM transactions (#1255)
* reduce the number of STM transactions

* reduce STM more

* refactor

* remove more
2024-08-09 16:38:46 +01:00
Evgeny Poberezkin 7d8457263b 6.0.0.7 2024-08-08 13:15:01 +01:00
Evgeny fc6b9c0e1b agent: stop statistics and cleanup when chat is stopped (#1251) 2024-08-07 13:14:09 +01:00
Evgeny Poberezkin a76e15fd77 6.0.0.6 2024-08-06 12:39:11 +01:00
Evgeny a017047c52 smp server: fix race when client is marked as subscribed after it is disconnected, preventing its GC (#1250)
* smp server: fix race when client is marked as subscribed after it is disconnected, preventing its GC

* refactor
2024-08-06 08:17:38 +01:00
Evgeny f4d1a33ed8 agent: retry loop that resumes subscriptions as soon as agent is moved to foreground, suspend retry loops while agent is suspended (#1249)
* agent: retry loop that resumes subscriptions as soon as agent is moved to foreground, suspend retry loops while agent is suspended

* reset retry enterval when moving to foreground

* account for network state too

* simplify

* typo

* simplify
2024-08-05 15:09:46 +01:00
Evgeny Poberezkin 03ea151be5 agent: support additional messages for connection in message batches (#1247)
* agent: support additional messages for connection in message batches

* fix, test

* test 2 connections

* use TVar instead of TMVar for GHC 8.10.7
2024-07-30 17:12:29 +01:00
Evgeny Poberezkin 83f8622b23 6.0.0.5 2024-07-29 21:02:22 +01:00
Evgeny Poberezkin 3753379ae4 agent: support dababase access with high priority for more responsive user actions (#1248)
* agent: support dababase access with high priority for more responsive user actions

* rename, export

* fix
2024-07-29 18:57:16 +01:00
Evgeny Poberezkin 2de16cfae8 6.0.0.4 2024-07-28 15:03:32 +01:00
Evgeny Poberezkinandspaced4ndy 5fa3c149e9 smp server: fix server-info, additional stats, allow expiring inactive clients which have prohibit subscriptions only (iOS NSE clients) (#1237)
* smp server: fix server-info

* fix

* faster saving messages

* remove comment

* move ProhibitSub out of TVar

* subscription stats

* stabilize test

* order

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* more notification stats

* count ntf stats

* update server-info

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-07-25 16:06:40 +01:00
Evgeny Poberezkin 9093c7b120 agent, ntf server: only mark subscriptions as pending that were created by the disconnected client (#1242)
* ntf server: only mark subscriptions as pending if the disconnected client is current

* add sessionId to subscribed queue

* add sessionId to subscriptions in ntf server agent

* fix
2024-07-25 13:07:28 +01:00
Evgeny Poberezkin 47ae921b86 use strict Map (#1241) 2024-07-23 08:27:54 +01:00
spaced4ndyandEvgeny Poberezkin c605156302 agent: getAgentSubsTotal api (#1238)
* agent: getAgentSubsTotal api

* move

* export

* count sessions

* simplify

* simplify 2

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-07-22 15:43:38 +01:00
Evgeny Poberezkin 051bf38bc7 agent: remove delays in notification processing, add notification stats (#1235)
* agent: remove delays in notification processing, add notification stats

* do not replace token after failed check

* more stats

* refactor

* fix

* backwards compatible JSON parsing

* retry deleting subscription on temporary error

* remove attempts to get multiple notification messages

* fix JSON decoding to be backwards compatible
2024-07-22 15:42:34 +01:00
Evgeny Poberezkin 8423c636a8 rename queueId to entId (can be message queue, proxied session and data blob ID to be added) (#1240) 2024-07-22 12:01:38 +01:00
Evgeny Poberezkin cc5732f41f smp server: stats for commands retrieving messages when notifications are received (#1236)
* more stats

* fix stats
2024-07-20 14:10:10 +01:00
spaced4ndy 7565ddd91c agent: initialize ratchet on processing confirmation to support decryption of messages received before allowConnection; return SndQueueSecured from joinConnection, acceptContact (#1233) 2024-07-18 19:54:14 +04:00
Evgeny Poberezkin 0de596dbcf 6.0.0.3 2024-07-18 11:05:53 +01:00
Evgeny Poberezkin e59a098e66 smp server: remove subscriptions from the clients when queues are deleted (#1234)
* smp server: remove subscriptions from the clients when queues are deleted

* remove ntf subscriptions, update stats

* add GET stats
2024-07-18 10:59:48 +01:00
Evgeny Poberezkin 8d56b0ba85 agent: allow starting only XFTP sending workers (#1232)
* agent: allow starting only XFTP sending workers

* use Bool param

* flip bool

* update
2024-07-17 13:01:17 +01:00
Evgeny Poberezkin b40d55c358 6.0.0.2 2024-07-15 20:30:39 +01:00
Evgeny Poberezkin 784d36d691 socks mode encoding (#1231)
* socks mode encoding

* test delays

* test delay

* unused import
2024-07-15 20:28:24 +01:00
Evgeny Poberezkin 1bdfc8ae00 6.0.0.1 2024-07-14 23:21:14 +01:00
Evgeny Poberezkin 291039159f ntf server: use SOCKS proxy to connect to onion-only SMP servers (#1229)
* ntf server: use SOCKS proxy to connect to onion-only SMP servers

* fix test
2024-07-14 23:19:02 +01:00
Evgeny Poberezkin d4fa0af350 ntf: additional tests for token registration when server and device are restarted (#1230)
* ntf: additional tests for token registration when server and device are restarted

* test response timeouts
2024-07-14 17:57:34 +01:00
Evgeny Poberezkin 492d2f86bc smp server: additional control port commands to monitor server state (#1228)
* smp server: additional control port commands to monitor server state

* fix

* space
2024-07-13 22:34:10 +01:00
Evgeny Poberezkin 8dd54ced0e agent: retry proxied command on NO_SESSION error, to prevent failure of proxied interactive commands (like joining connection) (#1227) 2024-07-13 10:06:48 +01:00
Evgeny Poberezkin e56bd0b47b agent: add known servers (#1225)
* agent: add known servers

* test delays

* ServerCfg

* json encoding

* enabledServerCfg

* checkUserServers
2024-07-12 12:41:55 +01:00
spaced4ndy ff2b00a029 agent: change ProxyClientError json encoding (#1226) 2024-07-11 19:27:04 +04:00
Evgeny Poberezkin 017469b2de 6.0.0.0 2024-07-09 13:56:02 +01:00
Evgeny Poberezkin 26979ff6b5 smp server: simplify client subscriptions (#1223) 2024-07-09 08:36:03 +01:00
Evgeny Poberezkin 21abc5cabe smp server: reduce the number of threads by delivering message to subscription when it is sent (#1222)
* smp server: reduce the number of threads by delivering message to subscription when it is sent

* test delay

* test delay
2024-07-08 23:12:01 +01:00
Evgeny Poberezkin 6e76221e07 agent: fix possible dead lock between sending and receiving messages, stress test for message delivery (#1224)
* agent: fix possible dead lock between sending and receiving messages, stress test for message delivery

* deliver events after the lock is released

* delayed delivery in command processing too

* tests: increase message expiration time
2024-07-08 21:47:42 +01:00
Evgeny Poberezkin 743676421d ntf server: simplify and optimize subscriptions in server agent (#1219)
* increase queue size

* simplify

* refactor to optimize memory usage and performance

* comment

* refactor

* test delays
2024-07-07 21:17:12 +01:00
Evgeny Poberezkin 9d0774a58e agent: add queue information (#1217)
* agent: add queue information to "debug delivery" response

* fix test

* rename

* encodings
2024-07-03 19:32:27 +01:00
Evgeny Poberezkin ce732c0efb agent: enable fast handshake (revert #1215) (#1216)
* Revert "agent: disable fast handshake (#1215)"

This reverts commit aa1d8d6c8b.

* remove import

* test delays
2024-07-03 18:05:27 +01:00
Evgeny Poberezkin 1ac0068d22 Merge commit 'f392ce0a9355cd3883400906ae6c361b77ca46ea' into stable 2024-07-03 14:09:54 +01:00
spaced4ndyandEvgeny Poberezkin ae8e1c5e9a agent: servers stats improvements, fixes (#1208)
* agent: reset stats startedAt time in memory

* getAgentSubsSummary

* change sub counting

* ack statistics

* add import

* instance

* Revert "instance"

This reverts commit 1f63740d56.

* Revert "add import"

This reverts commit ef72df8014.

* modify sub counting

* modify conn creation counting

* use int64

* file size stats

* remove import

* ack err counting

* conn del stats

* format

* new data

* add data

* toKB

* restore connCompleted

* use Int for counts

* use rq from scope

* remove getAgentSubsSummary

* fix connCompleted

* fix

* revert disabling stats

* use srv from scope

* combine ack stats

* modify

* comment

* count subs

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-07-03 12:36:15 +01:00
Evgeny Poberezkin f392ce0a93 5.8.2.0 2024-07-02 13:55:28 +01:00
Evgeny Poberezkin aa1d8d6c8b agent: disable fast handshake (#1215) 2024-07-02 13:40:37 +01:00
Evgeny Poberezkin ae4325d0e7 Merge pull request #1209 from simplex-chat/fast-connection
fast handshake protocol
2024-07-02 08:13:37 +01:00
Evgeny Poberezkin da5f669133 remove diagram 2024-07-01 23:34:31 +01:00
Evgeny Poberezkin 5c839b40c9 Merge branch 'master' into fast-connection 2024-07-01 21:14:32 +01:00
spaced4ndy a50e2e74a5 agent: disable saving stats (#1214)
* agent: disable stat saving

* disable migration

* schema

* disable restore
2024-07-01 21:06:38 +01:00
Evgeny Poberezkin 26cfad5e88 do not use sndSecure when rotating queue (#1213) 2024-07-01 13:42:46 +01:00
Evgeny Poberezkin a99ce6122c secure queue by sender via proxy (proxy SKEY command) (#1210)
* client: secure queue by sender via proxy (proxy SKEY command)

* agent and server: proxy SKEY command
2024-06-30 16:20:54 +01:00
Evgeny Poberezkin a6b542b301 Merge branch 'master' into fast-connection 2024-06-30 15:00:28 +01:00
Evgeny Poberezkin 6a54a58a0d agent: remove legacy statistics (#1211)
* agent: remove legacy statistics

* delays after disposeAgent

* delays

* enable all tests

* more delays
2024-06-30 12:50:42 +01:00
Evgeny Poberezkin 9ee684b0f4 rfc: faster handshake protocol (#1203)
* rfc: faster handshake protocol

* update

* 1 message

* SKEY

* use SKEY for both parties

* test

* update doc

* NEW command parameter

* add k=s param to queue URI

* fix

* add sndSecure field to queues

* make sender key non-optional in SndQueue (WIP, tests fail)

* fast handshake sometimes works (many tests fail)

* correctly handle SKEY retries, avoiding to re-generate the keys

* handle SKEY retries during async connection

* fix most tests (1 test fails)

* remove do

* fix contact requests encoding/tests

* export

* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages

* do not store sndSecure in store log if it is false to allow server downgrade

* add connection invitation encoding tests
2024-06-30 08:36:24 +01:00
spaced4ndyandEvgeny Poberezkin c788692687 agent: servers summary types, api (#1202)
* agent: servers summary types, api [wip]

* encoding

* export

* Revert "export"

This reverts commit cd9f315fe8.

* comment

* rename

* simplify types

* uncomment

* comment

* rework

* comment, exports

* save, restore stats wip

* remove

* rename

* save stats periodically

* sigint, sigterm experiments

* corrections

* remove some proxy stats

* increase stat

* proposed stats

* fields

* Revert "sigint, sigterm experiments"

This reverts commit f876fbd418.

* wip

* retries -> attempts

* errs

* fix

* other errs

* more stat tracking

* sub stats

* remove xftp successes stats

* xftp stats tracking

* revert

* revert

* refactor

* remove imports

* comment

* Revert "refactor"

This reverts commit 26c368d82a.

* Revert "revert"

This reverts commit 4c9e3753b5.

* Revert "revert"

This reverts commit 6f65644053.

* todos

* persistence

* rename, fix

* config

* comment

* add started at to summary

* delete stats on user deletion

* reset api

* move

* getAgentServersSummary collect state logic

* corrections

* corrections

* remove

* rework

* decrease contention

* update

* more stats

* count sentProxied

* count subs

* remove unused

* comment

* remove comment

* comment

* export

* refactor

* cleanup

* intervals

* refactor

* refactor2

* refactor3

* refactor4

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-25 09:42:59 +04:00
Alexander BondarenkoandEvgeny Poberezkin 9e7e0d102d smp-server: conserve resources (#1194)
* transport: force auth params, remove async wrapper

* stricter new messages

* bang more thunks

* style

* don't produce msgQuota unless requested

* strict

* refactor

* remove bangs

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-24 13:15:08 +01:00
d47c099ac9 docs: update protocol specifications (#1204)
* docs: update protocol specifications

* update SMP protocol (WIP)

* add proxy protocol commands and responses, amend envelope sizes in SMP protocol

* docs: update XFTP protocol (#1205)

* docs: update XFTP protocol

* commands

* fix table of contents, move sections

* add about ids

* download encryption

* qualities

* diagram

* crypto

* sending file diagram

* fix svg

* receiving file diagram

* update commands

* update handshake

* Add updated XRCP (#1207)

* add XRCP protocol

* add ToC

* update

* update

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

* add XFTP handshake description

* update agent protocol

* fast duplex connection in agent protocol

* update agent protocol

* update overview

* typos

* queue rotation, agent API, updates

* push notifications specification

* add XRCP threat model

* XFTP threat model

* update PQDR

* agent protocol end-to-end encryption

* versions

* remove TOC details

* update

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-06-23 22:53:45 +01:00
Evgeny Poberezkin e371a4a62d Merge branch 'master' into stable 2024-06-16 07:58:05 -07:00
sh 8a3b72458f readme: update compiling from source instructions (#1190) 2024-06-16 07:43:22 -07:00
Evgeny Poberezkin f616cb7b3e 5.8.1.0 2024-06-16 07:36:38 -07:00
spaced4ndyandEvgeny Poberezkin 4bbadffa37 agent: api to reconnect single server (#1198)
* agent: api to reconnect single server

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-16 06:27:25 -07:00
Evgeny PoberezkinandAlexander Bondarenko 388d77b61a ntf stats on SMP server (#1197)
* log undelivered notifications

* type

* add counters and encoding

* rename

* diff

* style

* style2

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-06-15 20:05:45 -07:00
Alexander Bondarenko 6597f6f0ed tests: add proxy connection refused test (#1200) 2024-06-15 19:27:58 -07:00
M. Sarmad QadeerandEvgeny Poberezkin 7008b08031 smp-server: fix layout (#1196)
* smp-server: fix layout

* ws

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-15 12:52:41 -07:00
spaced4ndyandEvgeny Poberezkin bb1d31e459 remote ctrl: differentiate RCP error when trying to connect to unknown host identity (#1195)
* remote ctrl: differentiate RCP error when trying to connect to unknown host identity

* rename

* refactor

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2024-06-10 17:42:51 +04:00
sh 369efc24c2 scripts: update install/update scripts (#1192) 2024-06-08 08:12:16 +01:00
spaced4ndy da620c388a xftp: start chunk upload after all chunks are prepared (#1191) 2024-06-06 15:50:15 +04:00
spaced4ndy 3c0cd7efcc agent: separate type for agent file errors (#1185) 2024-06-05 18:44:32 +04:00
Evgeny Poberezkin 3d605310ed agent: remove protocol encodings and agent TCP server (#1189)
* rfc: remove agent protocol encodings

* agent: remove protocol encodings and agent TCP server

* update

* remove unused code

* remove

* move tests

* add delay to tests

* stabilize test

* test

* more delays

* reduce delays

* enable all tests

* delays

* style
2024-06-05 14:34:40 +01:00
Evgeny Poberezkin 3dab330480 use throwE instead of throwError (#1187)
* use throwE instead of throwError

* test delay
2024-06-05 11:20:50 +01:00
Evgeny Poberezkin 44b8f265ae 5.8.0: changelog 2024-06-03 10:01:59 +01:00
Evgeny Poberezkin 2e4f507919 5.8.0.10 2024-05-31 22:20:30 +01:00
Evgeny Poberezkin d28b17e787 xftp server: send HTTP2 error as timeout error to the client so it is treated as temporary (#1186)
* xftp server: log file reception error

* report HTTP2 error as timeout error

* reduce timeout to 5 min

* process timeout error in protocol response

* log warning on timeout/HTTP2 error
2024-05-31 22:18:28 +01:00
Evgeny Poberezkin e1017e2a7f 5.8.0.9 2024-05-31 14:25:57 +01:00
Evgeny Poberezkin 1ff91d888f Merge branch 'stable' 2024-05-31 14:25:19 +01:00
Evgeny Poberezkin 0b5ab3a374 5.7.6.0 2024-05-31 14:23:21 +01:00
Evgeny Poberezkin 8ed54b33e0 agent: report correct errors from xftp handshake so they are treated as temporary (#1184)
* agent: report correct errors from xftp handshake so they are treated as temporary

* disable slow servers test

* remove comments

* all tests

* remove duplicate functions
2024-05-31 13:01:13 +01:00
Evgeny Poberezkin 3a3a84c58c server: log proxy connection errors 2024-05-31 12:20:29 +01:00
Evgeny Poberezkin d12ea92055 agent: report correct errors from xftp handshake so they are treated as temporary (#1184)
* agent: report correct errors from xftp handshake so they are treated as temporary

* disable slow servers test

* remove comments

* all tests

* remove duplicate functions
2024-05-31 09:47:47 +01:00
Evgeny Poberezkin 6e4067dc0c add string encodings for SMPProxyMode and SMPProxyFallback 2024-05-31 09:16:00 +01:00
Evgeny Poberezkinandspaced4ndy 88f1b727e0 SMP protocol extension to debug subscribed SMP queues (#1181)
* SMP protocol extension to debug subscribed SMP queues

* fix, test

* corrections

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* increase delays

* increase timeout

* delay

* delay

* enable all tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-05-30 18:49:43 +01:00
spaced4ndy 97a953550f agent: getAgentQueuesInfo (#1180) 2024-05-30 14:21:29 +04:00
Evgeny Poberezkin 39b3b5a25e 5.8.0.8 2024-05-29 13:19:10 +01:00
Alexander BondarenkoandEvgeny Poberezkin 15f0bb9e79 tcp-server: recover from accept errors (#1179)
* tcp-server: recover from accept errors

* log

* warn

* where

* retry

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-29 13:18:00 +01:00
5c2c88315a SMP server information (#1072)
* SMP server information

* fix tests

* country codes

* smp-server: serve contact and link pages from static files (#1084)

* smp-server: serve contact and link pages from static files

* generate index

* use params from ini

* render using ServerInformation

* tweak templates

* update

* fix some html

* smp-server: fix layout (#1097)

* smp-server: fix layout

* port fixes to link page

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* update server information page

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>

* update server info

* web: improve server info page design (#1166)

* web: improve server info page design

* web: fix font errors & some tags

* web: improve contact & invitation page layout and header

* update

* remove unused files/css

* cleanup

* fix link page

* remove unused font links

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* show contact address as is

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>
2024-05-29 11:30:42 +01:00
Evgeny Poberezkin 0f663bd569 5.8.0.7 2024-05-29 08:09:27 +01:00
Evgeny Poberezkin 63f5e76f9c agent: treat absent proxy session as a temporary error to retry sending (#1178)
* agent: treat absent proxy session as a temporary error to retry sending

* enable all tests
2024-05-29 08:06:01 +01:00
Evgeny Poberezkin ee052a454e 5.8.0.6 2024-05-28 15:33:55 +01:00
Evgeny Poberezkin e55ec07fe2 server: log stats for QUOTA and other errors (#1177)
* server: log stats for QUOTA errors

* fix test

* more stats

* remove duplicate column
2024-05-28 15:32:41 +01:00
Evgeny Poberezkin 199f85ec62 agent: send MWARN on QUOTA errors (#1176)
* agent: send MWARN on QUOTA errors

* fix tests
2024-05-28 11:56:57 +01:00
Evgeny Poberezkin 4a96dbf871 server: preprocess proxy commands that will not be connecting to network to reduce concurrency, do not wait for destination relay responses before processing the next command (#1174)
* server: preprocess proxy commands that will not be connecting to network to reduce concurrency

* implementation

* tests

* increase proxy client concurrency

* simplify

* refactor

* refactor2

* rename

* refactor3

* fix 8.10.7
2024-05-28 09:38:47 +01:00
Evgeny Poberezkin c8b2bb2ae1 agent: process message sent in expired response to ACK (it will increase duplicates count) (#1175) 2024-05-28 08:35:43 +01:00
Evgeny Poberezkin ab7b350521 agent: prevent sending not-batched client commands once requests time out (#1173) 2024-05-27 14:55:04 +01:00
Evgeny Poberezkin df35c50b99 5.7.5.0 2024-05-25 11:10:14 +01:00
Evgeny Poberezkin bd67844169 5.8.0.5 2024-05-24 14:26:02 +01:00
Evgeny Poberezkin 470dc74391 ntf server: do not persist server connection errors 2024-05-24 14:25:05 +01:00
Evgeny Poberezkin 2ff5f5a832 agent: add context to CMD error (#1167)
* agent: add context to CMD error

* tests, more warnings

* fix tests

* log TBQueue sizes

* log locks

* more logs

* log sendMessagesB

* fix test

* log length

* refactor

* remove logging

* revert lock scope change

* cleanup

* add string error to A_PROHIBITED

* remove

* remove test limitations

* language
2024-05-24 14:13:01 +01:00
Evgeny Poberezkin e7a73a4c89 Merge branch 'stable' 2024-05-23 22:04:06 +01:00
Evgeny Poberezkin 6309f92c68 agent: fail if non-unique connection IDs are passed to sendMessages (#1170) 2024-05-23 22:01:57 +01:00
Evgeny Poberezkin 875a56b75a Merge branch 'stable' 2024-05-23 16:48:11 +01:00
Alexander Bondarenko 984394d906 core: remove MonadUnliftIO ExceptT orphans (#1169) 2024-05-23 16:44:00 +01:00
Alexander Bondarenko 5d38ad03af tests: add proxy stress tests (#1163)
* tests: add proxy stress tests

* organize benches

* add agent tests

* move prints to logNote

* fix stuck agent tests
2024-05-23 15:34:25 +01:00
Alexander Bondarenko f6bb105536 utils: remove MonadError usage (#1168) 2024-05-23 12:47:30 +01:00
Evgeny Poberezkin 6c86aa302f 5.8.0.4 2024-05-22 16:13:26 +01:00
Evgeny Poberezkin 2b09ada392 agent: reduce interval for storing server connection errors to 3 seconds 2024-05-22 14:36:54 +01:00
Evgeny Poberezkin 8097df5540 agent: persist server connection error (#1165)
* agent: persist server connection

* comment, refactor

* fix tests, reduce interval for ntf server

* cleanup

* 0
2024-05-22 13:25:49 +01:00
Evgeny Poberezkin 5bff2f5537 Merge branch 'stable' 2024-05-21 23:42:49 +01:00
Evgeny Poberezkin 769e54db76 5.7.4.1 2024-05-21 22:58:58 +01:00
Evgeny Poberezkin f50589b31a agent: remove external timeout to resubscribe (#1164)
* agent: remove external timeout to resubscribe

* liftIO

* fix tests
2024-05-21 22:52:22 +01:00
Evgeny Poberezkin 2f2a3acdc0 5.8.0.3 2024-05-21 14:51:11 +01:00
Alexander BondarenkoandEvgeny Poberezkin f50fa5c60b smp-server: limit concurrency in proxy command processing (#1162)
* smp: put client proxy command processing threads under a shared semaphore

* add LIMITS.max_proc_threads to server config

* rename to PROXY.client_concurrency

* retry on strictly greater than max concurrency

* set default to 16

* rename

* fix limit

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-21 12:53:33 +01:00
Evgeny Poberezkin e3f5d244c1 5.8.0.2 2024-05-20 17:31:08 +01:00
Alexander BondarenkoandEvgeny Poberezkin f89d715a99 smp server: add proxy stats (#1157)
* smp-server: add proxy counters

* count simplex.im messages

* update

* fix

* get own servers from INI

* remove export

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-20 17:07:33 +01:00
Alexander BondarenkoandEvgeny Poberezkin 8fe18c4f6d core: use catMaybes to collect optional replies (#1161)
* core: use catMaybes to collect optional replies

* style

* rollback agent changes and add a note

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-20 15:48:21 +01:00
spaced4ndy 2e5433676e xftp: check proxy before downloading from unknown server (#1102)
* xftp: check proxy before downloading from unknown server

* corrections

* remove import

* tests

* comment

* tests

* don't wrap into internal error

* fix tests
2024-05-20 14:14:04 +01:00
Evgeny Poberezkin 8b21f7ef2a agent: aggregate multiple expired subscription responses into a single UP event (#1160)
* agent: aggregate multiple expired subscription responses into a single UP event

* clean up

* refactor processing of expired responses

* refactor

* refactor 2

* refactor unexpectedResponse
2024-05-20 07:56:51 +01:00
Evgeny Poberezkin 7a15ea59c9 Merge branch 'stable' 2024-05-19 07:58:00 +01:00
Evgeny Poberezkin 1bb6a5c43b agent: do not increase network activity interval while offline (#1159)
* agent: do not increase network activity interval while offline

* test
2024-05-19 07:50:47 +01:00
Evgeny Poberezkin 1116aeeea1 5.8.0.1 2024-05-17 15:39:20 +01:00
Evgeny Poberezkin 077ca9c046 Merge branch 'stable' 2024-05-17 15:38:06 +01:00
Evgeny Poberezkin 33f6d2f1da agent: optimize waiting for user network to avoid contention for state updates from multiple threads (#1155)
* haskell magic

* update

* agent: optimize waiting for user network with TChan

* clean up

* Int64

* use TVar

* cleanup

* fix

* testing

* update
2024-05-17 15:12:05 +01:00
spaced4ndy b33e8f4370 agent: reconnect xftp clients (#1156) 2024-05-17 14:23:59 +01:00
Alexander Bondarenko 426f47c805 smp: use session vars for reconnecting small agent (#1152)
* smp: use session vars for reconnecting small agent

* process errors

* split session and protocol functions

* add active flag to agent

* actually invoke agent shutdown

* close proxy agent too

* restore stopping ntf subscribers
2024-05-16 19:06:27 +01:00
Evgeny Poberezkin 9af3297997 Merge pull request #1151 from simplex-chat/ep/remember-error
proxy: remember server connection error for some time
2024-05-15 21:12:52 +01:00
Evgeny Poberezkin 16d79da73e config 2024-05-15 15:15:59 +01:00
Evgeny Poberezkin 6d19033375 proxy: remember server connection error for some time 2024-05-15 12:06:42 +01:00
Evgeny Poberezkin 2d2cc86bd8 Merge pull request #1095 from simplex-chat/proxy
SMP proxy feature branch
2024-05-14 21:35:51 +01:00
Evgeny Poberezkin f51cf1deac agent: use MVar for DB connection for more fair connection distribution between threads (#1147) 2024-05-14 20:04:51 +01:00
spaced4ndy 00a2dde727 Merge branch 'master' into proxy 2024-05-14 10:41:06 +04:00
Evgeny Poberezkin 762909ce33 5.8.0.0 2024-05-13 20:35:46 +01:00
Evgeny Poberezkin 512afa1e2b agent: count received duplicate messages (#1148)
* agent: count received duplicate messages

* count total too

* names

* fix

* tuple
2024-05-13 15:16:20 +01:00
Evgeny Poberezkin c4c983348f Merge branch 'master' into proxy 2024-05-13 14:13:52 +01:00
sh 969951d963 actions: ignore uploading GHC 8.10.7 binaries (#1064)
In other words, finally upload 22.04 binaries.
2024-05-13 10:46:37 +01:00
Evgeny Poberezkin 4455b8bd0e agent: do not throw exception when command is created for deleted connection (#1150)
* agent: do not throw exception when command is created for deleted connection

* convert database busy/locked to critical alert
2024-05-13 08:10:40 +01:00
Evgeny Poberezkin 66c916dbb3 proxy: increase client timeout for proxied commands (#1145) 2024-05-12 21:12:01 +01:00
Evgeny PoberezkinandAlexander Bondarenko 91cc48aabe agent: do not mark subscriptions on expired sessions as active, mark delayed subscriptions as active on the same session, do not cancel sending expired commands (#1127)
* agent: do not mark subscriptions on expired sessions as active, do mark delayed subscriptions as active on the same session, SUBOK response in the next SMP protocol version

* client: prevent sub actions from zombie sessions (#1122)

* client: prevent sub actions from zombie sessions

* error handling

* add AERR to pass background errors to client

* switch to activeClientSession

* put closeClient under activeClientSession

* rename

* remove AERR, do not skip processing

* move check and state update to one transaction

* catch extra UPs

* fix

* check queue is still pending before making it active

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* do not forward agent error

* revert not expiring sending subs

* fixes

* track subscription responses better

* add pending connection

* Revert "revert not expiring sending subs"

This reverts commit 4310a69391.

* do not expire sending commands

* rename

* fix race

* function

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-12 17:47:08 +01:00
Evgeny Poberezkin 1339a8da11 5.7.4.0 2024-05-11 23:39:28 +01:00
Evgeny Poberezkin 103ae06d55 agent: remove critical error on subscription timeout (#1146) 2024-05-11 23:38:27 +01:00
Evgeny Poberezkin 8516b0dd5b proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions (#1144)
* proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions

* rename

* inline

* comment

* use correct server version when encoding forwarded commands
2024-05-11 17:11:28 +01:00
Evgeny Poberezkin 9f72b249b4 Merge branch 'master' into proxy 2024-05-11 10:25:35 +01:00
Evgeny Poberezkin acc7faea11 5.7.3.1 2024-05-10 22:24:00 +01:00
Evgeny PoberezkinandAlexander Bondarenko 727fd8b8f5 server: more efficient response to batched subscriptions (#1141)
* server: more efficient response to batched subscriptions

* add sndMsgQ for interleaving messages with replies

* remove redundant liftIO

* refactor

* refactor2

* rename

* fix

* diff

* remove comment

* remove comment

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-10 22:19:11 +01:00
Evgeny Poberezkin dc111437fd 5.7.3.0 2024-05-10 15:15:43 +01:00
Evgeny Poberezkin b7afb725fd proxy: send MWARN event to user on server version or host more errors (#1140)
* proxy: include delivery path in SENT event

* send MWARN event to user on server version or host more errors

* Revert "proxy: include delivery path in SENT event"

This reverts commit 5c476718ec.
2024-05-10 10:55:19 +01:00
Evgeny Poberezkin b48215d341 proxy: additional configuration for SOCKS proxy usage in SMP proxy client (#1138)
* proxy: additional configuration for SOCKS proxy usage in SMP proxy client

* update
2024-05-09 15:36:02 +01:00
Evgeny Poberezkin 5cafd9d5c4 server: more efficient responses to batch subscriptions (#1137)
* server: more efficient responses to batch subscriptions

* comments

* comment

* enable tests

* LogError
2024-05-09 09:20:57 +01:00
Evgeny Poberezkin 817fd77332 Merge branch 'master' into proxy 2024-05-08 23:00:53 +01:00
Evgeny Poberezkin b27f126bab include server version range in transport handle (#1135)
* include server version range in transport handle

* xftp handshake

* remove coment

* simplify

* comments
2024-05-08 23:00:00 +01:00
Evgeny Poberezkin ea21b296fd agent: reset error count and do not report errors when consequitive timeouts happen while offline (#1136)
* agent: reset error count and do not report errors when consequitive timeouts happen while offline

* refactor

* comment
2024-05-08 15:33:51 +01:00
spaced4ndy 3f57d54832 xftp: catch exceptions in chunk download (#1133) 2024-05-08 16:57:04 +04:00
Evgeny Poberezkin b40654c95d update agent to v7/v2 SMP/NTF protocol versions (#997)
* update agent to v7/v2 SMP/NTF protocol versions
2024-05-08 13:05:06 +01:00
Evgeny Poberezkin 6d471b8be2 Merge branch 'master' into proxy 2024-05-08 00:14:06 +01:00
Alexander BondarenkoandEvgeny Poberezkin 7a07076277 transport: require ALPN for extended handshakes (#1134)
* transport: require ALPN for extended handshakes

* fix 8.10 build

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-08 00:06:09 +01:00
Evgeny Poberezkin a70f492f4d proxy: fallback to direct connection if destination relay does not support proxy protocol (#1132)
* proxy: fallback to direct connection if destination relay does not support proxy protocol

* move version to TransportError, refactor
2024-05-07 13:37:40 +01:00
Alexander BondarenkoandEvgeny Poberezkin c85f6a2f0e proxy: reporting errors (#1108)
* smp-proxy: iron out errors

* treat proxy timeouts as temporary

* update errors

* proxy errors (missing encoding)

* update

* enable tests

* update

* update

* fix

* fix

* simplify

* test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-06 22:00:42 +01:00
Evgeny Poberezkin f22daf5cb6 Merge branch 'master' into proxy 2024-05-06 15:26:11 +01:00
Evgeny Poberezkin 93fd424f86 5.7.2.0 2024-05-05 17:13:26 +01:00
Evgeny Poberezkin e13b0df539 client: remove TLS handshake timeout (#1129)
* client: remove TLS handshake timeout

* remove comment
2024-05-05 17:05:51 +01:00
Evgeny Poberezkin ee8e4067b0 agent: prepare connection record before joining to prevent race conditions (#1128)
* agent: prepare connection record before joining to prevent race conditions

* prepare connection for contact address as well

* clean up
2024-05-05 12:12:19 +01:00
Alexander BondarenkoandEvgeny Poberezkin 0e205e70ad add TRcvQueues tests (#1117)
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-03 23:39:00 +01:00
Evgeny Poberezkin b586a6e90a client: removed concurrency limit when waiting for subscription results (#1126) 2024-05-03 22:16:52 +01:00
Evgeny Poberezkin 259f6fcc3b Merge branch 'stable' 2024-05-02 16:24:32 +01:00
Evgeny Poberezkin 8d8010a62a 5.7.1.0 2024-05-02 16:22:55 +01:00
Evgeny Poberezkin c5941b790b client: increase timeout for SOCKS connection, increase timeout for direct connection (#1123) 2024-05-02 16:00:58 +01:00
Evgeny Poberezkin 60403955c0 5.7.0.4 2024-05-02 16:00:35 +01:00
Evgeny Poberezkinandspaced4ndy 9e49c289b4 upgrade SMP/NTF servers to v7/v2 protocol versions (#996)
* upgrade SMP/NTF servers to v7/v2 protocol versions

* 5.6.0.0

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-05-02 16:00:03 +01:00
Evgeny Poberezkin 6f83273318 client: increase timeout for SOCKS connection, increase timeout for direct connection (#1123) 2024-05-02 15:14:01 +01:00
Evgeny Poberezkin fb5d54e0a2 Merge branch 'master' into proxy 2024-05-01 12:52:04 +01:00
Alexander BondarenkoandEvgeny Poberezkin 0d2c1d3c01 transport: reduce ping traffic (#1118)
* transport: reduce ping traffic

* make pings opt-in, enable automatically with SUB commands

* fix reduced delays

* enable pings on MSG too

* rename pingErrorCount

* check timeout counter even when not sending pings

* clean up

* reset timeout error count on any event

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-05-01 10:38:33 +01:00
Evgeny PoberezkinandAlexander Bondarenko 6d60de2429 proxy: agent implementation (#1106)
* proxy: agent implementation

* revert change

* update rfc

* test stuck subscription mock

* store proxy sessions inside SMP client var

* rename

* create and use proxy session

* tests

* return proxy in SENT event

* rename, more tests

* rename

* more tests

* remove comment

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-05-01 08:48:33 +01:00
Evgeny Poberezkin 323cdc412e 5.7.0.4 2024-05-01 00:56:33 +01:00
Evgeny Poberezkinandspaced4ndy 66eccbecb3 upgrade SMP/NTF servers to v7/v2 protocol versions (#996)
* upgrade SMP/NTF servers to v7/v2 protocol versions

* 5.6.0.0

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-05-01 00:51:08 +01:00
Evgeny Poberezkin 935d2e25df 5.7.0.3 2024-04-25 10:22:18 +01:00
Alexander Bondarenko b931c1be37 utils: remove zstd contexts (#1116) 2024-04-25 09:33:22 +01:00
Evgeny Poberezkin daa866f333 5.7.0.2 2024-04-22 18:59:33 +01:00
Evgeny Poberezkin fe28e02be7 agent: make version independent of PQ enqryption support (#1114)
* agent: make version independent of PQ enqryption support

* remove comment
2024-04-22 13:40:24 +01:00
Evgeny Poberezkin 1612a7e2c7 agent: reduce sizes for per-queue e2e encryption (#1113) 2024-04-22 09:25:03 +01:00
Evgeny Poberezkin b08314722d 5.7.0.1 2024-04-21 22:58:15 +01:00
Evgeny Poberezkin 2347b82b47 adjust timeouts and delayed response errors (#1112) 2024-04-21 22:57:07 +01:00
Evgeny Poberezkin 3d40393ae8 5.7.0.0 2024-04-20 18:20:18 +01:00
Alexander BondarenkoandEvgeny Poberezkin b98fdb672d transport: increase client timeouts, don't send command after timeout (#1110)
* transport: fix client handshake timeouts

* fix handshake timeout

* skip sending requests for timed out responses

* expose batch concurrency as PClient field

* move to NetworkConfig

* remove Request on timeout

* use record

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-20 18:17:48 +01:00
Evgeny Poberezkin 3ba3172aaf xftp: enable ALPN in XFTP server (#1109) 2024-04-20 09:15:33 +01:00
Evgeny Poberezkin 2d1609f222 update envelope sizes for proxied messages, remove unnecessary proxy-relay encryption padding (#1107)
* update envelope sizes for proxied messages

* remove unnecessary padding from proxy-relay encryption
2024-04-19 20:24:25 +01:00
Alexander BondarenkoandEvgeny Poberezkin 58ede38bf4 put smp errors into proxy wrappers (#1103)
* put smp errors into proxy wrappers

* use substring in PROXY UNEXPECTED error

* fix encoding

* revert String encoding, discard invalid errors in QC

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-19 14:58:15 +01:00
Evgeny Poberezkin 7712070cb3 Merge branch 'master' into proxy 2024-04-18 23:33:06 +01:00
Evgeny Poberezkin c00c223f3b remove (or make optional) client key from handshakes (#1104)
* remove (or make optional) client key from handshakes

* remove comment
2024-04-18 22:43:49 +01:00
a3b229f668 SMP proxy: low level client and server implementation (#1096)
* SMP proxy: low level client and server implementation

* SMP proxy: server implementation (#1098)

* wip

* PRXY command

* progress

* SMP Proxy: client-level implementation (#1101)

* buildable

* encode messages

* update pkey

* fix queue types

* wrap SEND in proxy lookup

* WIP proxy client

* WIP

* post-rebase fixes

* encode something with something

* cleanup

* update

* fix nonce/corrId in batchingTests

* WIP: dig into createSMPProxySession

* agent

* test progress

* pass the test

* parameterize transport handle with transport peer to include server certificate (#1100)

* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>

* cleanup

* add 2-server test

* remove subsumed test

* checkCredentials for BrokerMsg

* skip batching tests

* remove userId param

* remove agent changes

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>

* remove unused type

* icrease test timeout

* reduce transport block

* envelope sizes

* don't fork unless have proxied commands to process

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-18 22:35:17 +01:00
Evgeny Poberezkin 8d7e0b1d09 Merge branch 'master' into proxy 2024-04-17 17:46:39 +01:00
2f43b43225 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-17 17:46:22 +01:00
ad4b5b6b71 parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate

* include server certificate into THandle

* load server chain and sign key

* fix key type

* fix for 8.10

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
2024-04-15 13:47:48 +01:00
Evgeny Poberezkin 98eb2742bc Merge branch 'master' into proxy 2024-04-13 18:51:17 +01:00
Evgeny Poberezkin ebb75ced12 extract SessionVar from AgentClient to reuse (#1099) 2024-04-13 18:33:12 +01:00
Evgeny Poberezkin 5e783396e0 5.6.2.2: v5.6.2 2024-04-12 23:14:20 +01:00
Evgeny Poberezkin b741b65700 use online status in network information (#1094)
* use online status in network information

* test

* refactor

* flip condition

* fix
2024-04-12 22:22:24 +01:00
Alexander BondarenkoandEvgeny Poberezkin 875ddd80d6 SMP proxy: protocol (#954)
* WIP: proxy-related types

* test plan

* buildable with stubs

* add auth test

* update protocol

* fix

* update rfc

* update protocol/types

* disable test

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-12 18:04:29 +01:00
Alexander BondarenkoandEvgeny Poberezkin 036b7523a5 xftp: matrix test for new versions (#1093)
* xftp: matrix test for new versions

* manual matrix

* more, faster

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-12 13:28:16 +01:00
spaced4ndyandEvgeny Poberezkin 3b1dd6dff9 agent: notify about available quota (#1087)
* agent: notify about available quota

* rename

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-11 09:38:12 +01:00
Evgeny Poberezkin 2e44ec5083 fix message expiration test (#1092) 2024-04-11 08:52:15 +01:00
Evgeny Poberezkin fbc25b171e xftp: reply ERR AUTH when file is removed from server storage (#1091) 2024-04-10 23:02:48 +01:00
Evgeny Poberezkinandspaced4ndy 8fa9ed6317 wait for user network availability (#1085)
* ghc-options

* wait for user network availability

* test

* update

* comment

* refactor

* slow config

* line

* waitForUserNetwork in xftp and ntf workers

* refactor

* refactor with registerDelay

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-04-10 21:50:05 +01:00
Alexander Bondarenko 91cf6841e0 http2: fix client setup (#1090)
* http2: cancel client action on setup timeout

* ignore incompatible server keys
2024-04-10 21:00:39 +01:00
Evgeny PoberezkinandAlexander Bondarenko 1219446996 dispose agent instances, fix tests, (#1089)
* dispose agent instances in tests

* fix quota test

* tests: fix tests with -threaded (#1088)

* fix some tests

* match RTS opts with apps

* less verbose rts stats

* enable sqlite extended error codes

* clean up

* unfocus

* remove extendedErrorCode

It's actually setExtendedResultCodes, which isn't yet available.

* diff

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* fix switch test

* fix

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-04-10 19:34:02 +01:00
Evgeny Poberezkin b994fd9f0f use sendRequest for HTTP2 2024-04-09 21:14:10 +01:00
Alexander BondarenkoandEvgeny Poberezkin 4c20ff6d00 xftp: negotiate protocol with ALPN (#1047)
* xftp: negotiate protocol with ALPN

* add RFC

* add handshake implementation

* implement extended handshake

* enable authentication

* update rfc

* Apply suggestions from code review

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* cleanup

* discard trailing data

* cleanup diff

* use find

* rename

* refactor

* add x509 tests

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-09 15:03:40 +01:00
spaced4ndyandEvgeny Poberezkin 791368c7be export lock function (#1073)
* export lock functions

* rename with tick

* simplify

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-04 15:37:31 +01:00
Alexander Bondarenko a4cfcfcc85 ntf-server: fix test token handling (#1083)
* use formatted logger

* fix DeviceToken parser for apns_null

* exclude PPNull from token stats
2024-04-04 14:46:40 +01:00
Evgeny Poberezkin 6bc4f6c94e 5.6.2.1 2024-04-03 10:04:41 +01:00
Evgeny Poberezkin 84b8c8417b revert base64 change (#1081) 2024-04-03 09:27:36 +01:00
Alexander Bondarenko be9a84905f servers: read stores by chunks (#1080) 2024-04-01 19:42:30 +01:00
Evgeny Poberezkin bfd532e833 5.6.2.0 2024-04-01 11:33:22 +01:00
Alexander BondarenkoandEvgeny Poberezkin a698f9a0c4 xftp-server: run expiration checks on start (#1042)
* xftp-server: run expiration checks on start

* rename

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-04-01 11:29:19 +01:00
39bb804fab xftp: fix repeated replica creation if it was in uploaded status (#1079)
* test with failing files (in progress)

* print

* add replica uploading state

* Revert "add replica uploading state"

This reverts commit 7068213aa6.

* <=

* fix

* prints

* test no redundancy

* all tests no redundancy

* revert delay

* refactor

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2024-04-01 13:37:35 +04:00
Evgeny PoberezkinandAlexander Bondarenko 6ded721daa remove monad typeclasses to reduce overhead (#1077)
* remove monad typeclasses to reduce overhead

* remove unliftIO

* StrictData

* inline

* optional agent port

* avoid MonadUnliftIO instance (#1078)

* avoid MonadUnliftIO instance

* simpler liftError'

* rename

* narrow down instance

* revert

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* logServer

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
2024-03-31 20:50:35 +01:00
450 changed files with 90951 additions and 16991 deletions
+44
View File
@@ -0,0 +1,44 @@
name: 'Set Swap Space'
description: 'Add moar swap'
branding:
icon: 'crop'
color: 'orange'
inputs:
swap-size-gb:
description: 'Swap space to create, in Gigabytes.'
required: false
default: '10'
runs:
using: "composite"
steps:
- name: Swap space report before modification
shell: bash
run: |
echo "Memory and swap:"
free -h
echo
swapon --show
echo
- name: Set Swap
shell: bash
run: |
export SWAP_FILE=$(swapon --show=NAME | tail -n 1)
echo "Swap file: $SWAP_FILE"
if [ -z "$SWAP_FILE" ]; then
SWAP_FILE=/opt/swapfile
else
sudo swapoff $SWAP_FILE
sudo rm $SWAP_FILE
fi
sudo fallocate -l ${{ inputs.swap-size-gb }}G $SWAP_FILE
sudo chmod 600 $SWAP_FILE
sudo mkswap $SWAP_FILE
sudo swapon $SWAP_FILE
- name: Swap space report after modification
shell: bash
run: |
echo "Memory and swap:"
free -h
echo
swapon --show
echo
+250 -56
View File
@@ -10,62 +10,25 @@ on:
pull_request:
jobs:
build:
name: build-${{ matrix.os }}-${{ matrix.ghc }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-20.04
platform_name: 20_04-x86-64
ghc: "8.10.7"
- os: ubuntu-20.04
platform_name: 20_04-x86-64
ghc: "9.6.3"
- os: ubuntu-22.04
platform_name: 22_04-x86-64
ghc: "9.6.3"
# =============================
# Create release
# =============================
# Create release, but only if it's triggered by tag push.
# On pull requests/commits push, this job will always complete.
maybe-release:
runs-on: ubuntu-latest
steps:
- name: Clone project
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/checkout@v3
- name: Setup Haskell
uses: haskell-actions/setup@v2
with:
ghc-version: ${{ matrix.ghc }}
cabal-version: "3.10.1.0"
- name: Cache dependencies
uses: actions/cache@v2
with:
path: |
~/.cabal/store
dist-newstyle
key: ${{ matrix.os }}-${{ hashFiles('cabal.project', 'simplexmq.cabal') }}
- name: Build
shell: bash
run: cabal build --enable-tests
- name: Test
timeout-minutes: 30
shell: bash
run: cabal test --test-show-details=direct
- name: Prepare binaries
if: startsWith(github.ref, 'refs/tags/v')
shell: bash
run: |
mv $(cabal list-bin smp-server) smp-server-ubuntu-${{ matrix.platform_name}}
mv $(cabal list-bin ntf-server) ntf-server-ubuntu-${{ matrix.platform_name}}
mv $(cabal list-bin xftp-server) xftp-server-ubuntu-${{ matrix.platform_name}}
mv $(cabal list-bin xftp) xftp-ubuntu-${{ matrix.platform_name}}
- name: Build changelog
if: startsWith(github.ref, 'refs/tags/v') && matrix.os == 'ubuntu-20.04'
id: build_changelog
uses: mikepenz/release-changelog-builder-action@v1
if: startsWith(github.ref, 'refs/tags/v')
uses: simplex-chat/release-changelog-builder-action@v5
with:
configuration: .github/changelog_conf.json
failOnError: true
@@ -75,8 +38,8 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Create release
if: startsWith(github.ref, 'refs/tags/v') && matrix.os == 'ubuntu-20.04' && matrix.ghc == '9.6.3'
uses: softprops/action-gh-release@v1
if: startsWith(github.ref, 'refs/tags/v')
uses: simplex-chat/action-gh-release@v2
with:
body: |
See full changelog [here](https://github.com/simplex-chat/simplexmq/blob/master/CHANGELOG.md).
@@ -86,10 +49,241 @@ jobs:
prerelease: true
files: |
LICENSE
smp-server-ubuntu-${{ matrix.platform_name}}
ntf-server-ubuntu-${{ matrix.platform_name}}
xftp-server-ubuntu-${{ matrix.platform_name}}
xftp-ubuntu-${{ matrix.platform_name}}
fail_on_unmatched_files: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# =============================
# Main build job
# =============================
build:
name: "ubuntu-${{ matrix.os }}-${{ matrix.arch }}, GHC: ${{ matrix.ghc }}"
needs: maybe-release
env:
apps: "smp-server xftp-server ntf-server xftp"
runs-on: ${{ matrix.runner }}
services:
postgres:
image: postgres:15
env:
POSTGRES_HOST_AUTH_METHOD: trust # Allows passwordless access
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
# Maps tcp port 5432 on service container to the host
- 5432:5432
strategy:
fail-fast: false
matrix:
include:
- os: 22.04
os_underscore: 22_04
arch: x86-64
runner: "ubuntu-22.04"
ghc: "8.10.7"
should_run: ${{ !(github.ref == 'refs/heads/stable' || startsWith(github.ref, 'refs/tags/v')) }}
- os: 22.04
os_underscore: 22_04
arch: x86-64
runner: "ubuntu-22.04"
ghc: "9.6.3"
should_run: true
- os: 24.04
os_underscore: 24_04
arch: x86-64
runner: "ubuntu-24.04"
ghc: "9.6.3"
should_run: true
- os: 22.04
os_underscore: 22_04
arch: aarch64
runner: "ubuntu-22.04-arm"
ghc: "9.6.3"
should_run: true
- os: 24.04
os_underscore: 24_04
arch: aarch64
runner: "ubuntu-24.04-arm"
ghc: "9.6.3"
should_run: true
steps:
- name: Clone project
if: matrix.should_run == true
uses: actions/checkout@v3
- name: Set up Docker Buildx
if: matrix.should_run == true
uses: simplex-chat/docker-setup-buildx-action@v3
- name: Setup swap
if: matrix.ghc == '8.10.7' && matrix.should_run == true
uses: ./.github/actions/swap
with:
swap-size-gb: 20
- name: Install PostgreSQL 15 client tools
if: matrix.os == '22.04' && matrix.should_run == true
shell: bash
run: |
# Import the repository signing key
sudo install -d /usr/share/postgresql-common/pgdg
sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
# Add the PostgreSQL APT repository
sudo sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# Update repository and install postgresql tools
sudo apt update
sudo apt -y install postgresql-client-15
- name: Build and cache Docker image
if: matrix.should_run == true
uses: simplex-chat/docker-build-push-action@v6
with:
context: .
load: true
file: Dockerfile.build
tags: build/${{ matrix.os }}:latest
build-args: |
TAG=${{ matrix.os }}
GHC=${{ matrix.ghc }}
- name: Cache dependencies
if: matrix.should_run == true
uses: actions/cache@v4
with:
path: |
~/.cabal/store
dist-newstyle
key: ubuntu-${{ matrix.os }}-${{ matrix.arch }}-ghc${{ matrix.ghc }}-${{ hashFiles('cabal.project', 'simplexmq.cabal') }}
- name: Start container
if: matrix.should_run == true
shell: bash
run: |
docker run -t -d \
--device /dev/fuse \
--cap-add SYS_ADMIN \
--security-opt apparmor:unconfined \
--name builder \
-v ~/.cabal:/root/.cabal \
-v /home/runner/work/_temp:/home/runner/work/_temp \
-v ${{ github.workspace }}:/project \
build/${{ matrix.os }}:latest
- name: Build smp-server (postgresql) and tests
if: matrix.should_run == true
shell: docker exec -t builder sh -eu {0}
run: |
chmod -fR 777 ~/.cabal ./dist-newstyle || :; git config --global --add safe.directory '*'
cabal clean
cabal update
cabal build --jobs=$(nproc) --enable-tests -fserver_postgres
mkdir -p /out
for i in smp-server simplexmq-test; do
bin=$(find /project/dist-newstyle -name "$i" -type f -executable)
chmod +x "$bin"
mv "$bin" /out/
done
strip /out/smp-server
- name: Copy simplexmq-test from container
if: matrix.should_run == true
shell: bash
run: |
docker cp builder:/out/simplexmq-test .
- name: Copy smp-server (postgresql) from container and prepare it
if: startsWith(github.ref, 'refs/tags/v') && matrix.should_run == true
id: prepare-postgres
shell: bash
run: |
name="smp-server-postgres-ubuntu-${{ matrix.os_underscore }}-${{ matrix.arch }}"
docker cp builder:/out/smp-server $name
path="${{ github.workspace }}/$name"
echo "bin=$path" >> $GITHUB_OUTPUT
hash="SHA2-256($name)= $(openssl sha256 $path | cut -d' ' -f 2)"
printf 'hash=%s' "$hash" >> $GITHUB_OUTPUT
- name: Build everything else (standard)
if: matrix.should_run == true
shell: docker exec -t builder sh -eu {0}
run: |
cabal build --jobs=$(nproc)
mkdir -p /out
for i in ${{ env.apps }}; do
bin=$(find /project/dist-newstyle -name "$i" -type f -executable)
strip "$bin"
chmod +x "$bin"
mv "$bin" /out/
done
- name: Copy binaries from container and prepare them
id: prepare-regular
if: startsWith(github.ref, 'refs/tags/v') && matrix.should_run == true
shell: bash
run: |
docker cp builder:/out .
printf 'bins<<EOF\n' > bins.output
printf 'hashes<<EOF\n' > hashes.output
for i in ${{ env.apps }}; do
name="$i-ubuntu-${{ matrix.os_underscore }}-${{ matrix.arch }}"
mv ./out/$i ./$name
path="${{ github.workspace }}/$name"
hash="SHA2-256($name)= $(openssl sha256 $path | cut -d' ' -f 2)"
printf '%s\n' "$path" >> bins.output
printf '%s\n\n' "$hash" >> hashes.output
done
printf 'EOF\n' >> bins.output
printf 'EOF\n' >> hashes.output
cat bins.output >> "$GITHUB_OUTPUT"
cat hashes.output >> "$GITHUB_OUTPUT"
- name: Upload binaries
if: startsWith(github.ref, 'refs/tags/v') && matrix.should_run == true
uses: simplex-chat/action-gh-release@v2
with:
append_body: true
prerelease: true
fail_on_unmatched_files: true
body: |
${{ steps.prepare-regular.outputs.hashes }}
${{ steps.prepare-postgres.outputs.hash }}
files: |
${{ steps.prepare-regular.outputs.bins }}
${{ steps.prepare-postgres.outputs.bin }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Test
if: matrix.should_run == true && matrix.arch == 'x86-64'
timeout-minutes: 120
shell: bash
env:
PGHOST: localhost
run: |
i=1
attempts=1
${{ (github.ref == 'refs/heads/stable' || startsWith(github.ref, 'refs/tags/v')) }} && attempts=3
while [ "$i" -le "$attempts" ]; do
if ./simplexmq-test; then
break
else
echo "Attempt $i failed, retrying..."
i=$((i + 1))
sleep 1
fi
done
if [ "$i" -gt "$attempts" ]; then
echo "All "$attempts" attempts failed."
exit 1
fi
+14 -6
View File
@@ -14,22 +14,22 @@ jobs:
matrix:
include:
- app: smp-server
app_port: 5223
app_port: "443 5223"
- app: xftp-server
app_port: 443
app_port: 443
steps:
- name: Clone project
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Log in to Docker Hub
uses: docker/login-action@v2
uses: simplex-chat/docker-login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Extract metadata for Docker image
id: meta
uses: docker/metadata-action@v4
uses: simplex-chat/docker-metadata-action@v5
with:
images: ${{ secrets.DOCKERHUB_USERNAME }}/${{ matrix.app }}
flavor: |
@@ -39,9 +39,17 @@ jobs:
type=semver,pattern=v{{major}}.{{minor}}
type=semver,pattern=v{{major}}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push Docker image
uses: docker/build-push-action@v4
uses: simplex-chat/docker-build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
build-args: |
APP=${{ matrix.app }}
+45
View File
@@ -0,0 +1,45 @@
name: Reproduce latest release
on:
workflow_dispatch:
schedule:
- cron: '0 2 * * *' # every day at 02:00 night
jobs:
reproduce:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Get latest release
shell: bash
run: |
curl --proto '=https' \
--tlsv1.2 \
-sSf -L \
'https://api.github.com/repos/simplex-chat/simplexmq/releases/latest' \
2>/dev/null | \
grep -i "tag_name" | \
awk -F \" '{print "TAG="$4}' >> $GITHUB_ENV
- name: Execute reproduce script
run: |
${GITHUB_WORKSPACE}/scripts/simplexmq-reproduce-builds.sh "$TAG" || :
- name: Check if build has been reproduced
env:
url: ${{ secrets.STATUS_SIMPLEX_WEBHOOK_URL }}
user: ${{ secrets.STATUS_SIMPLEX_WEBHOOK_USER }}
pass: ${{ secrets.STATUS_SIMPLEX_WEBHOOK_PASS }}
run: |
if [ -f "${GITHUB_WORKSPACE}/${TAG}-simplexmq/_sha256sums" ]; then
exit 0
else
curl --proto '=https' --tlsv1.2 -sSf \
-u "${user}:${pass}" \
-H 'Content-Type: application/json' \
-d '{"title": "👾 GitHub: Runner", "description": "⛔️ '"$TAG"' did not reproduce."}' \
"$url"
exit 1
fi
+1
View File
@@ -11,3 +11,4 @@ cabal.project.local~
.hpc/
*.tix
.coverage
@@ -0,0 +1,53 @@
# XFTPClientAgent Pattern
## TOC
1. Executive Summary
2. Changes: client.ts
3. Changes: agent.ts
4. Changes: test/browser.test.ts
5. Verification
## Executive Summary
Add `XFTPClientAgent` — a per-server connection pool matching the Haskell pattern. The agent caches `XFTPClient` instances by server URL. All orchestration functions (`uploadFile`, `downloadFile`, `deleteFile`) take `agent` as first parameter and use `getXFTPServerClient(agent, server)` instead of calling `connectXFTP` directly. Connections stay open on success; the caller creates and closes the agent.
`connectXFTP` and `closeXFTP` stay exported (used by `XFTPWebTests.hs` Haskell tests). The `browserClients` hack, per-function `connections: Map`, and `getOrConnect` are deleted.
## Changes: client.ts
**Add** after types section: `XFTPClientAgent` interface, `newXFTPAgent`, `getXFTPServerClient`, `closeXFTPServerClient`, `closeXFTPAgent`.
**Delete**: `browserClients` Map and all `isNode` browser-cache checks in `connectXFTP` and `closeXFTP`.
**Revert `closeXFTP`** to unconditional `c.transport.close()` (browser transport.close() is already a no-op).
`connectXFTP` stays exported (backward compat) but becomes a raw low-level function — no caching.
## Changes: agent.ts
**Imports**: replace `connectXFTP`/`closeXFTP` with `getXFTPServerClient`/`closeXFTPAgent` etc.
**Re-export** from agent.ts: `newXFTPAgent`, `closeXFTPAgent`, `XFTPClientAgent`.
**`uploadFile`**: add `agent: XFTPClientAgent` as first param. Replace `connectXFTP``getXFTPServerClient`. Remove `finally { closeXFTP }`. Pass `agent` to `uploadRedirectDescription`.
**`uploadRedirectDescription`**: change from `(client, server, innerFd)` to `(agent, server, innerFd)`. Get client via `getXFTPServerClient`.
**`downloadFile`**: add `agent` param. Delete local `connections: Map`. Replace `getOrConnect``getXFTPServerClient`. Remove finally cleanup. Pass `agent` to `downloadWithRedirect`.
**`downloadWithRedirect`**: add `agent` param. Same replacements. Remove try/catch cleanup. Recursive call passes `agent`.
**`deleteFile`**: add `agent` param. Same pattern.
**Delete**: `getOrConnect` function entirely.
## Changes: test/browser.test.ts
Create agent before operations, pass to upload/download, close in finally.
## Verification
1. `npx vitest --run` — browser round-trip test passes
2. No remaining `browserClients`, `getOrConnect`, or per-function `connections: Map` locals
3. `connectXFTP` and `closeXFTP` still exported (XFTPWebTests.hs compat)
4. All orchestration functions take `agent` as first param
+384
View File
@@ -1,3 +1,387 @@
# 6.4.4
Servers:
- fix server pages when source code is not specified.
- include commit SHA in printed version and in web page (#1608).
SMP server:
- support short SimpleX addresses in server information page (#1600).
- wrap all queries in transactions (#1603).
SMP agent:
- chat relay address type for short links (#1602).
- extend xrcp certificate validity 1 hour in the past, to allow out of sync clocks (#1601).
# 6.4.3
SMP agent:
- fix some connection errors by updating contact request server hosts to match server in short link (#1597).
SMP server:
- support short link URI as queue identifier in control port commands (#1596).
# 6.4.2
SMP server:
- fix memory leak when connection interrupts straight after client connects.
- do not include repeated queue blocking into stats/quota.
XFTP server:
- prometheus metrics
# 6.4.1
SMP protocol:
- create notification credentials via NEW command that creates the queue (#1586)
SMP server:
- control port session improvements (#1591)
- additional stat counter for ntf credentials created together with the queue (#1589)
# 6.4.0
SMP protocol (server/client):
- support associated queue data and short connection links (see [RFC](./rfcs/2025-03-16-smp-queues.md)).
- service certificates to optimize subscriptions.
SMP agent:
- support retries for interactive connection handshakes.
- use web port 443 by default for preset servers.
- use static RNG function to avoid creating dynamic C stubs when generating sntrup keys (it was detected as Dynamic Code Loading in GrapheneOS).
- different timeouts for interactive and background operations.
Ntf server:
- PostgreSQL storage.
- Prometheus metrics.
- use service certificates.
- fix repeat token registration.
# 6.3.2
Servers:
- enable store log by default (#1501).
SMP server:
- reduce memory usage (#1498)
SMP agent:
- handle client/agent version downgrades after connection was established (#1508).
# 6.3.1
Servers:
- handle ECONNABORTED error on client connections.
- reproducible builds.
- blocking records for content moderation.
- update script (simplex-servers-update) downloads scripts from the specified or the latest stable tag.
SMP server:
- support for PostgreSQL database for queue records for higher traffic servers.
- fix old clients sending messages to new servers (#1443)
- remove empty journals when opening message queues and expiring idle queues (#1456, #1458).
- additional start options (#1465):
- `maintenance` to run all start/stop operations without starting server.
- `skip-warnings` to ignore the last corrupted line in store log (can happen on abnormal termination).
Ntf server:
- record date of last token activity, to allow expiring inactive tokens.
- additional token invalidation reasons in logs.
SMP agent:
- store message sent to multiple connections only once, to reduce storage when sending to groups (#1453).
- encrypt messages on delivery, to reduce database writes (#1446).
- don't block method calls on congested sockets for better concurrency (#1454).
- check notification token status on client connection.
- option to skip SQLite vacuum on migrations.
# 6.3.0
SMP agent: fix joining connection after failure by using the same ratchet.
# 6.2.2
SMP server:
- add optional Prometheus metrics (#1411).
Build:
- remove three modules from client library.
# 6.2.0
Version 6.2.0.7
Build:
- client_library flag to build only used modules in the clients, remove package yaml
SMP server:
- journal storage for messages (BETA).
- prevent race condition when deleting queue and to avoid "orphan" messages (#1395).
SMP agent:
- support SMP and XFTP server roles (storage/proxy) and operators (#1343).
- treat blocked STM and other critical errors that offer restart as temporary for message delivery (#1405).
- fix inconsistent state after app restart while accepting contact request (#1412).
# 6.1.3
SMP server: fix restoring notification credentials.
# 6.1.2
Servers: more reliable restoring of state.
SMP server: reduced memory usage and faster start.
Notifications: compensate for iOS notifications being dropped by Apple while device is offline (#1378):
- Ntf server: send multiple SMP notifications in one iOS notification.
- Agent: get multiple messages for one iOS notification.
# 6.1.1
SMP:
- stop server faster (#1371)
- add STORE error (#1372)
# 6.1.0
Version 6.1.0.7
SMP server and client:
- transport block encryption (#1317).
Agent:
- batch and optimize iOS notifications processing (#1308, #1311, #1313, #1316, #1330, #1331, #1333, #1337, #1346).
- allow receiving multiple messages from single iOS notification (#1355, #1362).
- prepare connection to accept to avoid race condition with events (#1365).
- transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes (#1321).
Ntf server:
- control port (#1354).
- enable pings on ntf subscriptions, to resubscribe on reconnection (#1353).
SMP server:
- support multiple server ports (#1319).
- support serving HTTPS and SMP transport on the same port (#1326, #1327).
- persist iOS notifications to avoid losing them when Ntf server is offline (#1336, #1339, #1350).
- fix lost notification subscriptions (#1347).
- reject SKEY with different key earlier, at verification step (#1366).
- pass server information via CLI during server initialization (#1356).
- show version on server page (#1341).
- explicit graceful shutdown on SIGINT (#1360).
XRCP (remote access protocol):
- use SHA3-256 in hybrid key agreement (#1302).
- session encryption with forward secrecy (#1328).
# 6.0.5
SMP agent:
- support generic SOCKS proxy (without isolate-by-auth).
- reduce max message sizes
# 6.0.4
SMP server:
- better performance/memory: fewer map updates on re-subscriptions (#1297), split and reduce STM transactions (#1294)
- send DELD when subscribed queue is deleted (#1312)
- add created/updated/used date to queues to manage expiration (#1306)
XFTP server: truncate file creation time to 1 hour (#1310)
Servers:
- bind control port only to 127.0.0.1 for better security in case of firewall misconfiguration (#1280)
- reduce memory used for period stats (#1298)
Agent: process last notification from list (#1307)
- report receive file error with redirected file ID, when redirect is present (#1304)
- special error when deleted user record is not in database (#1303)
- fix race when sending a message to the deleted connection (#1296)
- support for multiple messages in a single notification
Ntf server:
- only use SOCKS proxy for servers without public address (#1314)
# 6.0.3
Agent:
- fix possible stuck queue rotation (#1290).
SMP server:
- batch END responses when subscribed client switches to reduce server and client traffic.
- reduce STM transactions for better performance.
- add stats for END events and for SUB/DEL event batches.
- remove "expensive" stats to save memory.
# 6.0.2
SMP agent:
- fix stuck connection commands when a server is not responding.
- store query errors, reduce slow query threshold to 1ms.
Notification server:
- reduce PING interval to 1 minute.
- fix subscriptions disabled on race condition (only mark subscriptions with END status when received via the active connection).
# 6.0.1
SMP agent:
- support changing user of the new connection.
- do not start delivery workers when there are no messages to deliver.
- enable notifications for all connections.
- combine database transactions when subscribing.
SMP server:
- safe compacting of store log.
- fix possible race when creating client that might lead to memory leak.
Dependencies: upgrade tls to 1.9
# 6.0.0
Version 6.0.0.8
Agent:
- enabled fast handshake support.
- batch-send multiple messages in each connection.
- resume subscriptions as soon as agent moves to foreground or as network connection resumes.
- "known" servers to determine whether to use SMP proxy.
- retry on SMP proxy NO_SESSION error.
- fixes to notification subscriptions.
- persistent server statistics.
- better concurrency.
SMP server:
- reduce threads usage.
- additional statistics.
- improve disabling inactive clients.
- additional control port commands for monitoring.
Notification server:
- support onion-only SMP servers.
# 5.8.2
Agent:
- fast handshake support (disabled).
- new statistics api.
SMP server:
- fast handshake support (SKEY command).
- minor changes to reduce memory usage.
# 5.8.1
Agent:
- API to reconnect one server.
- Better error handling of file errors and remote control connection errors.
- Only start uploading file once all chunks were registered on the servers.
SMP server:
- additional stats for sent message notifications.
- fix server page layout.
# 5.8.0
Version 5.8.0.10
SMP server and client:
- protocol extension to forward messages to the destination servers, to protect sending client IP address and transport session.
Agent:
- process timed out subscription responses to reduce the number of resubscriptions.
- avoid sending messages and commands when waiting for response timed out (except batched SUB and DEL commands).
- fix issue with stuck message reception on slow connection (when response to ACK timed out, and the new message was not processed until resubscribed).
- fix issue when temporary file sending or receiving error was treated as permanent.
SMP server:
- include OK responses to all batched SUB requests to reduce subscription timeouts.
XFTP server:
- report file upload timeout as TIMEOUT, to avoid delivery failure.
# 5.7.6
XFTP agent:
- treat XFTP handshake timeouts and network errors as temporary, to retry file operations.
# 5.7.5
SMP agent:
- fail if non-unique connection IDs are passed to sendMessages (to prevent client errors and deadlocks).
# 5.7.4
SMP agent:
- remove re-subscription timeouts (as they are tracked per operation, and could cause failed subscriptions).
- reconnect XFTP clients when network settings changes.
- fix lock contention resulting in stuck subscriptions on network change.
# 5.7.3
SMP/NTF protocol:
- add ALPN for handshake version negotiation, similar to XFTP (to preserve backwards compatibility with the old clients).
- upgrade clients to versions v7/v2 of the protocols.
SMP server:
- faster responses to subscription requests.
XFTP client:
- fix network exception during file download treated as permanent file error.
SMP agent:
- do not report subscription timeouts while client is offline.
# 5.7.2
SMP agent:
- fix connections failing when connecting via link due to race condition on slow network.
- remove concurrency limit when waiting for connection subscription.
- remove TLS timeout.
# 5.7.1
SMP agent:
- increase timeout for TLS connection via SOCKS
# 5.7.0
Version 5.7.0.4
_Please note_: the earliest SimpleX Chat clients supported by this version of the servers is 5.5.3 (released on February 11, 2024).
SMP server:
- increase max SMP protocol version to 7 (support for deniable authenticators).
NTF server:
- increase max NTF protocol version to 2 (support for deniable authenticators).
XFTP server:
- version handshake using ALPN.
SMP agent:
- increase timeouts for XFTP files.
- don't send commands after timeout.
- PQ encryption support.
# 5.6.2
Version 5.6.2.2.
SMP agent:
- Lower memory consumption (~20-25%).
- More stable XFTP file uploads and downloads.
- API to receive network connectivity changes from the apps.
- to reduce battery consumption: connection attempts interval growing to every 2 hours when app reports as offline.
- to reduce retries and traffic: 50% increased timeouts when on mobile network.
XFTP server:
- expire files on start.
- version negotiation based on TLS ALPN and handshake.
NTF server:
- reduced downtime by ~100x faster start time.
- exclude test tokens from statistics.
# 5.6.1
Version 5.6.1.0.
+31 -8
View File
@@ -1,15 +1,20 @@
ARG TAG=22.04
# syntax=docker/dockerfile:1.7.0-labs
ARG TAG=24.04
FROM ubuntu:${TAG} AS build
### Build stage
# Install curl and git and simplexmq dependencies
RUN apt-get update && apt-get install -y curl git build-essential libgmp3-dev zlib1g-dev llvm-12 llvm-12-dev libnuma-dev libssl-dev
RUN apt-get update && apt-get install -y curl git build-essential libgmp3-dev zlib1g-dev llvm-18 llvm-18-dev libnuma-dev libssl-dev
# Specify bootstrap Haskell versions
ENV BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3
ENV BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.1.0
ENV BOOTSTRAP_HASKELL_CABAL_VERSION=3.12.1.0
# Do not install Stack
ENV BOOTSTRAP_HASKELL_INSTALL_NO_STACK=true
ENV BOOTSTRAP_HASKELL_INSTALL_NO_STACK_HOOK=true
# Install ghcup
RUN curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh
@@ -21,26 +26,42 @@ ENV PATH="/root/.cabal/bin:/root/.ghcup/bin:$PATH"
RUN ghcup set ghc "${BOOTSTRAP_HASKELL_GHC_VERSION}" && \
ghcup set cabal "${BOOTSTRAP_HASKELL_CABAL_VERSION}"
COPY . /project
# Copy only the source code
COPY apps /project/apps/
COPY cbits /project/cbits/
COPY src /project/src/
COPY cabal.project Setup.hs simplexmq.cabal LICENSE /project
WORKDIR /project
# Debug
#ARG CACHEBUST=1
#ADD --chmod=755 https://github.com/MShekow/directory-checksum/releases/download/v1.4.6/directory-checksum_1.4.6_linux_amd64 /usr/local/bin/directory-checksum
#RUN directory-checksum --max-depth 2 .
# Set build arguments and check if they exist
ARG APP
ARG APP_PORT
RUN if [ -z "$APP" ] || [ -z "$APP_PORT" ]; then printf "Please spcify \$APP and \$APP_PORT build-arg.\n"; exit 1; fi
RUN if [ -z "$APP" ]; then printf "Please spcify \$APP build-arg.\n"; exit 1; fi
# Compile app
RUN cabal update
RUN cabal build exe:$APP
# Copy scripts
COPY scripts /project/scripts/
# Create new path containing all files needed
RUN mkdir /final
WORKDIR /final
# Strip the binary from debug symbols to reduce size
RUN bin=$(find /project/dist-newstyle -name "$APP" -type f -executable) && \
RUN bin="$(find /project/dist-newstyle -name "$APP" -type f -executable)" && \
mv "$bin" ./ && \
strip ./"$APP" &&\
mv /project/scripts/docker/entrypoint-"$APP" ./entrypoint
mv /project/scripts/docker/entrypoint-"$APP" ./entrypoint &&\
mv /project/scripts/main/simplex-servers-stopscript ./simplex-servers-stopscript
### Final stage
FROM ubuntu:${TAG}
@@ -53,6 +74,8 @@ COPY --from=build /final /usr/local/bin/
# Open app listening port
ARG APP_PORT
RUN if [ -z "$APP_PORT" ]; then printf "Please spcify \$APP_PORT build-arg.\n"; exit 1; fi
EXPOSE $APP_PORT
# simplexmq requires using SIGINT to correctly preserve undelivered messages and restore them on restart
+31
View File
@@ -0,0 +1,31 @@
# syntax=docker/dockerfile:1.7.0-labs
ARG TAG=24.04
FROM ubuntu:${TAG} AS build
### Build stage
ARG GHC=9.6.3
ARG CABAL=3.14.1.1
# Install curl, git and and simplexmq dependencies
RUN apt-get update && apt-get install -y curl libpq-dev git sqlite3 libsqlite3-dev build-essential libgmp3-dev zlib1g-dev llvm llvm-dev libnuma-dev libssl-dev
# Specify bootstrap Haskell versions
ENV BOOTSTRAP_HASKELL_GHC_VERSION=${GHC}
ENV BOOTSTRAP_HASKELL_CABAL_VERSION=${CABAL}
# Do not install Stack
ENV BOOTSTRAP_HASKELL_INSTALL_NO_STACK=true
ENV BOOTSTRAP_HASKELL_INSTALL_NO_STACK_HOOK=true
# Install ghcup
RUN curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh
# Adjust PATH
ENV PATH="/root/.cabal/bin:/root/.ghcup/bin:$PATH"
# Set both as default
RUN ghcup set ghc "${GHC}" && \
ghcup set cabal "${CABAL}"
WORKDIR /project
+36 -16
View File
@@ -1,6 +1,6 @@
# SimpleXMQ
[![GitHub build](https://github.com/simplex-chat/simplexmq/workflows/build/badge.svg)](https://github.com/simplex-chat/simplexmq/actions?query=workflow%3Abuild)
[![GitHub build](https://github.com/simplex-chat/simplexmq/actions/workflows/build.yml/badge.svg)](https://github.com/simplex-chat/simplexmq/actions/workflows/build.yml)
[![GitHub release](https://img.shields.io/github/v/release/simplex-chat/simplexmq)](https://github.com/simplex-chat/simplexmq/releases)
📢 SimpleXMQ v1 is released - with many security, privacy and efficiency improvements, new functionality - see [release notes](https://github.com/simplex-chat/simplexmq/releases/tag/v1.0.0).
@@ -116,7 +116,7 @@ On Linux, you can deploy smp and xftp server using Docker. This will download im
2. Run your Docker container.
- `smp-server`
You must change **your_ip_or_domain**. `-e "pass=password"` is optional variable to password-protect your `smp` server:
```sh
docker run -d \
@@ -129,7 +129,7 @@ On Linux, you can deploy smp and xftp server using Docker. This will download im
```
- `xftp-server`
You must change **your_ip_or_domain** and **maximum_storage**.
```sh
docker run -d \
@@ -149,8 +149,15 @@ On Linux, you can deploy smp and xftp server using Docker. This will download im
You can install and setup servers automatically using our script:
```sh
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/simplex-chat/simplexmq/stable/install.sh -o simplex-server-install.sh \
&& if echo 'b8cf2be103f21f9461d9a500bcd3db06ab7d01d68871b07f4bd245195cbead1d simplex-server-install.sh' | sha256sum -c; then chmod +x ./simplex-server-install.sh && ./simplex-server-install.sh; rm ./simplex-server-install.sh; else echo "SHA-256 checksum is incorrect!" && rm ./simplex-server-install.sh; fi
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/simplex-chat/simplexmq/stable/install.sh -o simplex-server-install.sh &&\
if echo '53fcdb4ceab324316e2c4cda7e84dbbb344f32550a65975a7895425e5a1be757 simplex-server-install.sh' | sha256sum -c; then
chmod +x ./simplex-server-install.sh
./simplex-server-install.sh
rm ./simplex-server-install.sh
else
echo "SHA-256 checksum is incorrect!"
rm ./simplex-server-install.sh
fi
```
### Build from source
@@ -180,7 +187,7 @@ On Linux, you can build smp server using Docker.
3. Run your Docker container.
- `smp-server`
You must change **your_ip_or_domain**. `-e "pass=password"` is optional variable to password-protect your `smp` server:
```sh
docker run -d \
@@ -193,7 +200,7 @@ On Linux, you can build smp server using Docker.
```
- `xftp-server`
You must change **your_ip_or_domain** and **maximum_storage**.
```sh
docker run -d \
@@ -208,14 +215,17 @@ On Linux, you can build smp server using Docker.
#### Using your distribution
1. Install [Haskell GHCup](https://www.haskell.org/ghcup/), GHC 8.10.7 and cabal:
1. Install dependencies and build tools (`GHC`, `cabal` and dev libs):
```sh
curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh
ghcup install ghc 8.10.7
ghcup install cabal
ghcup set ghc 8.10.7
ghcup set cabal
# On Ubuntu. Depending on your distribution, use your package manager to determine package names.
sudo apt-get update && apt-get install -y build-essential curl libffi-dev libffi7 libgmp3-dev libgmp10 libncurses-dev libncurses5 libtinfo5 pkg-config zlib1g-dev libnuma-dev libssl-dev
export BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3
export BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.3.0
curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh
ghcup set ghc "${BOOTSTRAP_HASKELL_GHC_VERSION}"
ghcup set cabal "${BOOTSTRAP_HASKELL_CABAL_VERSION}"
source ~/.ghcup/env
```
2. Build the project:
@@ -224,10 +234,20 @@ On Linux, you can build smp server using Docker.
git clone https://github.com/simplex-chat/simplexmq
cd simplexmq
git checkout stable
# On Ubuntu. Depending on your distribution, use your package manager to determine package names.
apt-get update && apt-get install -y build-essential libgmp3-dev zlib1g-dev
cabal update
cabal install
cabal build exe:smp-server exe:xftp-server
```
3. List compiled binaries:
`smp-server`
```sh
cabal list-bin exe:smp-server
```
`xftp-server`
```sh
cabal list-bin exe:xftp-server
```
- Initialize SMP server with `smp-server init [-l] -n <fqdn>` or `smp-server init [-l] --ip <ip>` - depending on how you initialize it, either FQDN or IP will be used for server's address.
+15
View File
@@ -0,0 +1,15 @@
{-# LANGUAGE TemplateHaskell #-}
module Web.Embedded where
import Data.FileEmbed (embedDir, embedFile)
import Simplex.Messaging.Server.Web (EmbeddedContent (..))
embeddedContent :: EmbeddedContent
embeddedContent =
EmbeddedContent
{ indexHtml = $(embedFile "apps/common/Web/static/index.html"),
linkHtml = $(embedFile "apps/common/Web/static/link.html"),
mediaContent = $(embedDir "apps/common/Web/static/media/"),
wellKnown = $(embedDir "apps/common/Web/static/.well-known/")
}
@@ -0,0 +1,49 @@
{
"applinks": {
"details": [
{
"appIDs": [
"5NN7GUYB6T.chat.simplex.app"
],
"components": [
{
"/": "/contact/*"
},
{
"/": "/contact"
},
{
"/": "/invitation/*"
},
{
"/": "/invitation"
},
{
"/": "/a/*"
},
{
"/": "/a"
},
{
"/": "/c/*"
},
{
"/": "/c"
},
{
"/": "/g/*"
},
{
"/": "/g"
},
{
"/": "/i/*"
},
{
"/": "/i"
}
]
}
]
}
}
@@ -0,0 +1,16 @@
[
{
"relation": [
"delegate_permission/common.handle_all_urls"
],
"target": {
"namespace": "android_app",
"package_name": "chat.simplex.app",
"sha256_cert_fingerprints": [
"5E:3E:DC:C2:00:FB:A8:D5:F4:88:F3:CA:4C:32:5B:05:78:C5:6A:9C:03:A1:CC:B5:92:9C:D7:5C:7E:57:E2:4D",
"3C:52:C4:FD:3C:AD:1C:07:C9:B0:0A:70:80:E3:58:FA:B9:FE:FC:B8:AF:5A:EC:14:77:65:F1:6D:0F:21:AD:85",
"AE:C1:95:DC:FD:46:14:BD:3A:91:EC:26:D1:D5:14:C8:75:71:C5:CC:8D:CF:48:08:3F:92:83:14:3C:A2:B9:A6"
]
}
}
]
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

@@ -0,0 +1,26 @@
<svg width="119" height="40" viewBox="0 0 119 40" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M8.44484 39.125C8.14016 39.125 7.84284 39.1211 7.54055 39.1143C6.91433 39.1061 6.28957 39.0516 5.67141 38.9512C5.095 38.8519 4.53661 38.6673 4.01467 38.4033C3.49751 38.1415 3.02582 37.7983 2.61767 37.3867C2.20361 36.98 1.85888 36.5082 1.59716 35.9902C1.33255 35.4688 1.14942 34.9099 1.05416 34.333C0.951281 33.7131 0.895621 33.0863 0.887656 32.458C0.881316 32.2471 0.873016 31.5449 0.873016 31.5449V8.44434C0.873016 8.44434 0.881856 7.75293 0.887706 7.5498C0.895332 6.92248 0.950669 6.29665 1.05324 5.67773C1.14868 5.09925 1.33194 4.53875 1.5967 4.01563C1.85746 3.49794 2.20027 3.02586 2.61184 2.61768C3.02294 2.20562 3.49614 1.8606 4.01418 1.59521C4.53492 1.33209 5.09225 1.14873 5.6675 1.05127C6.28769 0.949836 6.91462 0.894996 7.54301 0.88721L8.44533 0.875H111.214L112.127 0.8877C112.75 0.895099 113.371 0.94945 113.985 1.05029C114.566 1.14898 115.13 1.33362 115.656 1.59814C116.694 2.13299 117.539 2.97916 118.071 4.01807C118.332 4.53758 118.512 5.09351 118.606 5.66699C118.71 6.29099 118.768 6.92174 118.78 7.5542C118.783 7.8374 118.783 8.1416 118.783 8.44434C118.791 8.81934 118.791 9.17627 118.791 9.53613V30.4648C118.791 30.8281 118.791 31.1826 118.783 31.54C118.783 31.8652 118.783 32.1631 118.779 32.4697C118.768 33.0909 118.71 33.7104 118.608 34.3232C118.515 34.9043 118.333 35.4675 118.068 35.9932C117.805 36.5056 117.462 36.9733 117.053 37.3789C116.644 37.7927 116.172 38.1379 115.653 38.4014C115.128 38.6674 114.566 38.8527 113.985 38.9512C113.367 39.0522 112.742 39.1067 112.116 39.1143C111.823 39.1211 111.517 39.125 111.219 39.125L110.135 39.127L8.44484 39.125Z" fill="black"/>
<path d="M24.7689 20.3007C24.7796 19.466 25.0013 18.6477 25.4134 17.9217C25.8254 17.1957 26.4144 16.5858 27.1254 16.1486C26.6737 15.5035 26.0778 14.9725 25.3849 14.598C24.6921 14.2234 23.9215 14.0156 23.1343 13.991C21.455 13.8147 19.8271 14.9958 18.9714 14.9958C18.0991 14.9958 16.7816 14.0085 15.3629 14.0376C14.4452 14.0673 13.5509 14.3341 12.767 14.8122C11.9831 15.2903 11.3364 15.9632 10.89 16.7655C8.95597 20.1139 10.3986 25.035 12.2512 27.7416C13.1781 29.0669 14.2613 30.5474 15.6788 30.4949C17.0659 30.4374 17.5839 29.6104 19.2582 29.6104C20.917 29.6104 21.403 30.4949 22.8492 30.4615C24.3376 30.4374 25.2753 29.1303 26.1697 27.7924C26.8357 26.848 27.3481 25.8043 27.6881 24.6999C26.8234 24.3341 26.0855 23.722 25.5664 22.9397C25.0473 22.1574 24.7699 21.2396 24.7689 20.3007V20.3007Z" fill="white"/>
<path d="M22.0373 12.2109C22.8488 11.2367 23.2486 9.98451 23.1518 8.72028C21.9119 8.8505 20.7667 9.44306 19.9442 10.3799C19.5421 10.8376 19.2341 11.37 19.0378 11.9468C18.8416 12.5235 18.7609 13.1333 18.8005 13.7413C19.4206 13.7477 20.0341 13.6132 20.5948 13.3482C21.1555 13.0831 21.6487 12.6942 22.0373 12.2109Z" fill="white"/>
<path d="M42.3023 27.1396H37.5689L36.4322 30.4961H34.4273L38.9107 18.0781H40.9937L45.4771 30.4961H43.438L42.3023 27.1396ZM38.0591 25.5908H41.8111L39.9615 20.1435H39.9097L38.0591 25.5908Z" fill="white"/>
<path d="M55.1597 25.9697C55.1597 28.7832 53.6538 30.5908 51.3814 30.5908C50.8057 30.6209 50.2332 30.4883 49.7294 30.2082C49.2256 29.928 48.8109 29.5117 48.5327 29.0068H48.4897V33.4912H46.6313V21.4424H48.4302V22.9482H48.4644C48.7553 22.4458 49.1771 22.0316 49.6847 21.7497C50.1923 21.4679 50.7669 21.3289 51.3472 21.3476C53.645 21.3477 55.1597 23.1641 55.1597 25.9697ZM53.2495 25.9697C53.2495 24.1367 52.3023 22.9316 50.857 22.9316C49.437 22.9316 48.482 24.1621 48.482 25.9697C48.482 27.7939 49.437 29.0156 50.857 29.0156C52.3023 29.0156 53.2495 27.8193 53.2495 25.9697Z" fill="white"/>
<path d="M65.1245 25.9697C65.1245 28.7832 63.6187 30.5908 61.3462 30.5908C60.7706 30.6209 60.1981 30.4883 59.6943 30.2082C59.1905 29.928 58.7758 29.5117 58.4976 29.0068H58.4546V33.4912H56.5962V21.4424H58.395V22.9482H58.4292C58.7201 22.4458 59.1419 22.0316 59.6495 21.7497C60.1571 21.4679 60.7317 21.3289 61.312 21.3476C63.6099 21.3476 65.1245 23.164 65.1245 25.9697ZM63.2144 25.9697C63.2144 24.1367 62.2671 22.9316 60.8218 22.9316C59.4019 22.9316 58.4468 24.1621 58.4468 25.9697C58.4468 27.7939 59.4019 29.0156 60.8218 29.0156C62.2671 29.0156 63.2144 27.8193 63.2144 25.9697H63.2144Z" fill="white"/>
<path d="M71.7105 27.0361C71.8482 28.2676 73.0445 29.0761 74.6792 29.0761C76.2456 29.0761 77.3726 28.2675 77.3726 27.1572C77.3726 26.1933 76.6929 25.6162 75.0835 25.2207L73.4742 24.833C71.1939 24.2822 70.1353 23.2158 70.1353 21.4853C70.1353 19.3427 72.0025 17.871 74.6538 17.871C77.2778 17.871 79.0767 19.3427 79.1372 21.4853H77.2612C77.1489 20.246 76.1245 19.498 74.6274 19.498C73.1304 19.498 72.106 20.2548 72.106 21.3564C72.106 22.2343 72.7603 22.7509 74.3608 23.1464L75.729 23.4823C78.2769 24.0849 79.3355 25.1083 79.3355 26.9247C79.3355 29.248 77.4849 30.703 74.5415 30.703C71.7876 30.703 69.9282 29.2821 69.8081 27.036L71.7105 27.0361Z" fill="white"/>
<path d="M83.3462 19.2998V21.4424H85.0679V22.9141H83.3462V27.9053C83.3462 28.6807 83.6909 29.042 84.4478 29.042C84.6522 29.0384 84.8562 29.0241 85.0591 28.999V30.4619C84.7188 30.5255 84.373 30.5543 84.0269 30.5478C82.1939 30.5478 81.479 29.8593 81.479 28.1035V22.9141H80.1626V21.4424H81.479V19.2998H83.3462Z" fill="white"/>
<path d="M86.065 25.9697C86.065 23.1211 87.7427 21.3311 90.3589 21.3311C92.9839 21.3311 94.6539 23.1211 94.6539 25.9697C94.6539 28.8262 92.9927 30.6084 90.3589 30.6084C87.7261 30.6084 86.065 28.8262 86.065 25.9697ZM92.7603 25.9697C92.7603 24.0156 91.8648 22.8623 90.3589 22.8623C88.8531 22.8623 87.9585 24.0244 87.9585 25.9697C87.9585 27.9316 88.8531 29.0762 90.3589 29.0762C91.8648 29.0762 92.7603 27.9316 92.7603 25.9697H92.7603Z" fill="white"/>
<path d="M96.1861 21.4424H97.9585V22.9834H98.0015C98.1215 22.5021 98.4034 22.0768 98.8 21.7789C99.1966 21.481 99.6836 21.3287 100.179 21.3476C100.393 21.3469 100.607 21.3702 100.816 21.417V23.1553C100.546 23.0726 100.264 23.0347 99.981 23.043C99.711 23.032 99.4419 23.0796 99.192 23.1825C98.9422 23.2854 98.7176 23.4411 98.5336 23.639C98.3496 23.8369 98.2106 24.0723 98.1262 24.3289C98.0418 24.5856 98.0139 24.8575 98.0445 25.126V30.4961H96.1861L96.1861 21.4424Z" fill="white"/>
<path d="M109.384 27.8369C109.134 29.4805 107.534 30.6084 105.486 30.6084C102.852 30.6084 101.217 28.8437 101.217 26.0127C101.217 23.1729 102.861 21.3311 105.408 21.3311C107.913 21.3311 109.488 23.0518 109.488 25.7969V26.4336H103.093V26.5459C103.064 26.8791 103.105 27.2148 103.216 27.5306C103.326 27.8464 103.502 28.1352 103.732 28.3778C103.963 28.6203 104.242 28.8111 104.552 28.9374C104.861 29.0637 105.195 29.1226 105.529 29.1103C105.968 29.1515 106.409 29.0498 106.785 28.8203C107.162 28.5909 107.455 28.246 107.62 27.8369L109.384 27.8369ZM103.102 25.1348H107.628C107.645 24.8352 107.6 24.5354 107.495 24.2541C107.39 23.9729 107.229 23.7164 107.02 23.5006C106.812 23.2849 106.561 23.1145 106.283 23.0003C106.006 22.8861 105.708 22.8305 105.408 22.8369C105.105 22.8351 104.805 22.8933 104.525 23.008C104.245 23.1227 103.99 23.2918 103.776 23.5054C103.562 23.7191 103.392 23.973 103.276 24.2527C103.16 24.5323 103.101 24.8321 103.102 25.1348V25.1348Z" fill="white"/>
<path d="M37.8262 8.73101C38.2158 8.70305 38.6068 8.76191 38.9709 8.90335C39.335 9.04478 39.6632 9.26526 39.9318 9.54889C40.2004 9.83251 40.4026 10.1722 40.524 10.5435C40.6455 10.9148 40.6829 11.3083 40.6338 11.6959C40.6338 13.6021 39.6035 14.6979 37.8262 14.6979H35.6709V8.73101H37.8262ZM36.5977 13.854H37.7227C38.0011 13.8707 38.2797 13.825 38.5382 13.7204C38.7968 13.6158 39.0287 13.4548 39.2172 13.2493C39.4057 13.0437 39.546 12.7987 39.6279 12.5321C39.7097 12.2655 39.7311 11.9839 39.6904 11.708C39.7282 11.4332 39.7046 11.1534 39.6215 10.8887C39.5384 10.6241 39.3977 10.3811 39.2097 10.1771C39.0216 9.97322 38.7908 9.81341 38.5337 9.70917C38.2766 9.60494 37.9997 9.55885 37.7227 9.57422H36.5977V13.854Z" fill="white"/>
<path d="M41.6807 12.4443C41.6524 12.1484 41.6862 11.8499 41.7801 11.5678C41.8739 11.2857 42.0257 11.0264 42.2256 10.8064C42.4255 10.5864 42.6693 10.4107 42.9411 10.2904C43.213 10.1701 43.507 10.108 43.8042 10.108C44.1015 10.108 44.3955 10.1701 44.6673 10.2904C44.9392 10.4107 45.1829 10.5864 45.3828 10.8064C45.5828 11.0264 45.7345 11.2857 45.8284 11.5678C45.9222 11.8499 45.9561 12.1484 45.9278 12.4443C45.9566 12.7406 45.9232 13.0396 45.8296 13.3221C45.736 13.6046 45.5843 13.8644 45.3843 14.0848C45.1843 14.3052 44.9404 14.4814 44.6683 14.6019C44.3962 14.7225 44.1018 14.7847 43.8042 14.7847C43.5066 14.7847 43.2123 14.7225 42.9401 14.6019C42.668 14.4814 42.4241 14.3052 42.2241 14.0848C42.0241 13.8644 41.8725 13.6046 41.7789 13.3221C41.6853 13.0396 41.6518 12.7406 41.6807 12.4443V12.4443ZM45.0137 12.4443C45.0137 11.4683 44.5752 10.8975 43.8057 10.8975C43.0332 10.8975 42.5987 11.4683 42.5987 12.4444C42.5987 13.4282 43.0333 13.9946 43.8057 13.9946C44.5752 13.9946 45.0137 13.4243 45.0137 12.4443H45.0137Z" fill="white"/>
<path d="M51.5733 14.6978H50.6514L49.7207 11.3813H49.6504L48.7237 14.6978H47.8106L46.5694 10.1948H47.4707L48.2774 13.6308H48.3438L49.2696 10.1948H50.1221L51.0479 13.6308H51.1182L51.9209 10.1948H52.8096L51.5733 14.6978Z" fill="white"/>
<path d="M53.8536 10.1948H54.709V10.9102H54.7754C54.8881 10.6532 55.0781 10.4379 55.319 10.2941C55.5598 10.1503 55.8396 10.0852 56.1192 10.1079C56.3383 10.0914 56.5583 10.1245 56.7629 10.2046C56.9675 10.2847 57.1514 10.4098 57.3011 10.5706C57.4508 10.7315 57.5624 10.9239 57.6276 11.1337C57.6928 11.3436 57.7099 11.5654 57.6778 11.7827V14.6977H56.7891V12.0059C56.7891 11.2822 56.4746 10.9224 55.8174 10.9224C55.6687 10.9154 55.5202 10.9408 55.3821 10.9966C55.244 11.0524 55.1197 11.1375 55.0176 11.2458C54.9154 11.3542 54.838 11.4834 54.7904 11.6245C54.7429 11.7657 54.7265 11.9154 54.7422 12.0635V14.6978H53.8535L53.8536 10.1948Z" fill="white"/>
<path d="M59.0938 8.43701H59.9825V14.6978H59.0938V8.43701Z" fill="white"/>
<path d="M61.2178 12.4443C61.1895 12.1484 61.2234 11.8498 61.3172 11.5677C61.4111 11.2857 61.5629 11.0263 61.7629 10.8063C61.9628 10.5863 62.2065 10.4106 62.4784 10.2903C62.7503 10.17 63.0443 10.1079 63.3416 10.1079C63.6389 10.1079 63.9329 10.17 64.2047 10.2903C64.4766 10.4106 64.7203 10.5863 64.9203 10.8063C65.1203 11.0263 65.272 11.2857 65.3659 11.5677C65.4598 11.8498 65.4936 12.1484 65.4654 12.4443C65.4942 12.7406 65.4607 13.0396 65.3671 13.3221C65.2734 13.6046 65.1218 13.8644 64.9217 14.0848C64.7217 14.3052 64.4778 14.4814 64.2057 14.6019C63.9335 14.7224 63.6392 14.7847 63.3416 14.7847C63.044 14.7847 62.7496 14.7224 62.4775 14.6019C62.2053 14.4814 61.9614 14.3052 61.7614 14.0848C61.5614 13.8644 61.4097 13.6046 61.3161 13.3221C61.2225 13.0396 61.189 12.7406 61.2178 12.4443V12.4443ZM64.5508 12.4443C64.5508 11.4683 64.1123 10.8975 63.3428 10.8975C62.5703 10.8975 62.1358 11.4683 62.1358 12.4444C62.1358 13.4282 62.5704 13.9946 63.3428 13.9946C64.1123 13.9946 64.5508 13.4243 64.5508 12.4443H64.5508Z" fill="white"/>
<path d="M66.4009 13.4243C66.4009 12.6138 67.0044 12.1465 68.0757 12.0801L69.2954 12.0098V11.6211C69.2954 11.1455 68.981 10.877 68.3736 10.877C67.8775 10.877 67.5337 11.0591 67.4351 11.3775H66.5747C66.6656 10.604 67.3931 10.1079 68.4146 10.1079C69.5435 10.1079 70.1802 10.6699 70.1802 11.6211V14.6978H69.3247V14.065H69.2544C69.1117 14.292 68.9113 14.477 68.6737 14.6012C68.4361 14.7254 68.1697 14.7844 67.9019 14.772C67.7129 14.7916 67.5218 14.7715 67.341 14.7128C67.1603 14.6541 66.9938 14.5581 66.8524 14.4312C66.711 14.3042 66.5977 14.149 66.52 13.9756C66.4422 13.8022 66.4017 13.6144 66.4009 13.4243V13.4243ZM69.2954 13.0396V12.6631L68.1958 12.7334C67.5757 12.7749 67.2945 12.9859 67.2945 13.3828C67.2945 13.7881 67.646 14.0239 68.1295 14.0239C68.2711 14.0383 68.4142 14.024 68.5502 13.9819C68.6862 13.9398 68.8123 13.8708 68.9211 13.7789C69.0299 13.6871 69.1191 13.5743 69.1834 13.4473C69.2477 13.3203 69.2858 13.1816 69.2954 13.0396V13.0396Z" fill="white"/>
<path d="M71.3482 12.4444C71.3482 11.0215 72.0796 10.1201 73.2173 10.1201C73.4987 10.1072 73.778 10.1746 74.0226 10.3145C74.2671 10.4544 74.4667 10.661 74.5982 10.9101H74.6646V8.43701H75.5533V14.6978H74.7017V13.9863H74.6314C74.4898 14.2338 74.2832 14.4378 74.0339 14.5763C73.7847 14.7148 73.5023 14.7825 73.2173 14.772C72.0718 14.772 71.3482 13.8706 71.3482 12.4444ZM72.2662 12.4444C72.2662 13.3994 72.7164 13.9741 73.4693 13.9741C74.2183 13.9741 74.6812 13.3911 74.6812 12.4483C74.6812 11.5098 74.2134 10.9185 73.4693 10.9185C72.7212 10.9185 72.2661 11.4971 72.2661 12.4444H72.2662Z" fill="white"/>
<path d="M79.23 12.4443C79.2017 12.1484 79.2356 11.8499 79.3294 11.5678C79.4232 11.2857 79.575 11.0264 79.7749 10.8064C79.9749 10.5864 80.2186 10.4107 80.4904 10.2904C80.7623 10.1701 81.0563 10.108 81.3536 10.108C81.6508 10.108 81.9448 10.1701 82.2167 10.2904C82.4885 10.4107 82.7322 10.5864 82.9322 10.8064C83.1321 11.0264 83.2839 11.2857 83.3777 11.5678C83.4715 11.8499 83.5054 12.1484 83.4771 12.4443C83.5059 12.7406 83.4725 13.0396 83.3789 13.3221C83.2853 13.6046 83.1336 13.8644 82.9336 14.0848C82.7336 14.3052 82.4897 14.4814 82.2176 14.6019C81.9455 14.7225 81.6512 14.7847 81.3536 14.7847C81.0559 14.7847 80.7616 14.7225 80.4895 14.6019C80.2173 14.4814 79.9735 14.3052 79.7735 14.0848C79.5735 13.8644 79.4218 13.6046 79.3282 13.3221C79.2346 13.0396 79.2012 12.7406 79.23 12.4443V12.4443ZM82.563 12.4443C82.563 11.4683 82.1245 10.8975 81.355 10.8975C80.5826 10.8975 80.148 11.4683 80.148 12.4444C80.148 13.4282 80.5826 13.9946 81.355 13.9946C82.1245 13.9946 82.563 13.4243 82.563 12.4443Z" fill="white"/>
<path d="M84.6695 10.1948H85.5249V10.9102H85.5913C85.704 10.6532 85.894 10.4379 86.1349 10.2941C86.3757 10.1503 86.6555 10.0852 86.9351 10.1079C87.1542 10.0914 87.3742 10.1245 87.5788 10.2046C87.7834 10.2847 87.9673 10.4098 88.117 10.5706C88.2667 10.7315 88.3783 10.9239 88.4435 11.1337C88.5087 11.3436 88.5258 11.5654 88.4937 11.7827V14.6977H87.605V12.0059C87.605 11.2822 87.2906 10.9224 86.6333 10.9224C86.4846 10.9154 86.3361 10.9408 86.198 10.9966C86.06 11.0524 85.9356 11.1375 85.8335 11.2458C85.7314 11.3542 85.6539 11.4834 85.6064 11.6245C85.5588 11.7657 85.5424 11.9154 85.5581 12.0635V14.6978H84.6695V10.1948Z" fill="white"/>
<path d="M93.5152 9.07373V10.2153H94.4908V10.9639H93.5152V13.2793C93.5152 13.751 93.7095 13.9575 94.1519 13.9575C94.2651 13.9572 94.3783 13.9503 94.4908 13.937V14.6772C94.3312 14.7058 94.1695 14.721 94.0074 14.7226C93.0191 14.7226 92.6255 14.375 92.6255 13.5068V10.9638H91.9107V10.2153H92.6255V9.07373H93.5152Z" fill="white"/>
<path d="M95.7046 8.43701H96.5855V10.9185H96.6558C96.7739 10.6591 96.9691 10.4425 97.2148 10.2982C97.4605 10.1539 97.7448 10.0888 98.0288 10.1118C98.2467 10.1 98.4646 10.1364 98.6669 10.2184C98.8692 10.3004 99.0509 10.4261 99.199 10.5864C99.3471 10.7468 99.458 10.9378 99.5238 11.146C99.5896 11.3541 99.6086 11.5742 99.5796 11.7905V14.6978H98.69V12.0098C98.69 11.2905 98.355 10.9263 97.7271 10.9263C97.5744 10.9137 97.4207 10.9347 97.277 10.9878C97.1332 11.0408 97.0027 11.1247 96.8948 11.2334C96.7868 11.3421 96.7038 11.4732 96.6518 11.6173C96.5997 11.7614 96.5798 11.9152 96.5933 12.0679V14.6977H95.7047L95.7046 8.43701Z" fill="white"/>
<path d="M104.761 13.4819C104.641 13.8935 104.379 14.2495 104.022 14.4876C103.666 14.7258 103.236 14.8309 102.81 14.7847C102.513 14.7925 102.219 14.7357 101.946 14.6182C101.674 14.5006 101.43 14.3252 101.232 14.1041C101.034 13.8829 100.887 13.6214 100.8 13.3376C100.713 13.0537 100.689 12.7544 100.73 12.4605C100.691 12.1656 100.715 11.8656 100.801 11.581C100.888 11.2963 101.034 11.0335 101.231 10.8105C101.428 10.5874 101.671 10.4092 101.942 10.288C102.214 10.1668 102.509 10.1054 102.806 10.1079C104.059 10.1079 104.815 10.9639 104.815 12.3779V12.688H101.635V12.7378C101.621 12.9031 101.642 13.0694 101.696 13.2261C101.75 13.3829 101.837 13.5266 101.95 13.6481C102.062 13.7695 102.2 13.866 102.352 13.9314C102.504 13.9968 102.669 14.0297 102.835 14.0278C103.047 14.0533 103.262 14.0151 103.453 13.9178C103.644 13.8206 103.802 13.6689 103.906 13.4819L104.761 13.4819ZM101.635 12.0308H103.91C103.921 11.8796 103.9 11.7278 103.849 11.5851C103.798 11.4424 103.718 11.3119 103.614 11.2021C103.509 11.0922 103.383 11.0054 103.243 10.9472C103.103 10.8891 102.953 10.8608 102.801 10.8643C102.648 10.8623 102.495 10.8912 102.353 10.9491C102.21 11.0071 102.081 11.0929 101.972 11.2017C101.864 11.3104 101.778 11.4397 101.72 11.5821C101.662 11.7245 101.633 11.8771 101.635 12.0308H101.635Z" fill="white"/>
</svg>

After

Width:  |  Height:  |  Size: 16 KiB

+77
View File
@@ -0,0 +1,77 @@
(function () {
let complete = false
run()
window.onload = run
async function run() {
const connURIel = document.getElementById("conn_req_uri_text");
const mobileConnURIanchor = document.getElementById("mobile_conn_req_uri");
const connQRCodes = document.getElementsByClassName("conn_req_uri_qrcode");
console.log(connQRCodes);
if (complete || !connURIel || !mobileConnURIanchor || connQRCodes < 2) return
complete = true
let connURI = document.location.toString()
const parsedURI = new URL(connURI)
const path = parsedURI.pathname.split("/")
const len = path.length
const action = path[len - (path[len - 1] == "" ? 2 : 1)]
parsedURI.protocol = "https"
parsedURI.pathname = "/" + action
connURI = parsedURI.toString()
console.log("connection URI: ", connURI)
const hash = parsedURI.hash
const hostname = parsedURI.hostname
let appURI = "simplex:" + parsedURI.pathname
appURI += action.length > 1 // not short link
? hash
: !hash.includes("?") // otherwise add server hostname
? hash + "?h=" + hostname // no parameters
: !hash.includes("?h=") && !hash.includes("&h=")
? hash + "&h=" + hostname // no "h" parameter
: hash.replace(/([?&])h=([^&]+)/, `$1h=${hostname},$2`) // add as the first hostname to "h" parameter
mobileConnURIanchor.href = appURI
console.log("app URI: ", appURI)
connURIel.innerText = "/c " + connURI
for (const connQRCode of connQRCodes) {
try {
await QRCode.toCanvas(connQRCode, connURI, {
errorCorrectionLevel: "M",
color: {dark: "#062D56"}
});
connQRCode.style.width = "320px";
connQRCode.style.height = "320px";
} catch (err) {
console.error(err);
}
}
function contentCopyWithTooltip(parent) {
const content = parent.querySelector(".content");
const tooltip = parent.querySelector(".tooltiptext");
console.log(parent.querySelector(".content_copy"), 111)
console.log(parent)
const copyButton = parent.querySelector(".content_copy");
copyButton.addEventListener("click", copyAddress)
copyButton.addEventListener("mouseout", resetTooltip)
function copyAddress() {
navigator.clipboard.writeText(content.innerText || content.value);
tooltip.innerHTML = "Copied!";
}
function resetTooltip() {
tooltip.innerHTML = "Copy to clipboard";
}
}
function copyAddress() {
navigator.clipboard.writeText(connURI);
tooltipEl.innerHTML = "Copied!";
}
function resetTooltip() {
tooltipEl.innerHTML = "Copy to clipboard";
}
}
})();
Binary file not shown.

After

Width:  |  Height:  |  Size: 289 KiB

+372
View File
@@ -0,0 +1,372 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
width="560"
height="164"
version="1.1"
id="svg1048"
sodipodi:docname="f_droid.svg"
inkscape:version="1.1.2 (0a00cf5339, 2022-02-04, custom)"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<sodipodi:namedview
id="namedview1050"
pagecolor="#505050"
bordercolor="#ffffff"
borderopacity="1"
inkscape:pageshadow="0"
inkscape:pageopacity="0"
inkscape:pagecheckerboard="1"
showgrid="false"
inkscape:snap-bbox="true"
inkscape:bbox-paths="true"
inkscape:bbox-nodes="true"
inkscape:snap-bbox-edge-midpoints="true"
inkscape:snap-bbox-midpoints="true"
inkscape:snap-intersection-paths="true"
inkscape:object-paths="true"
inkscape:snap-smooth-nodes="true"
inkscape:snap-midpoints="true"
inkscape:snap-object-midpoints="true"
inkscape:snap-center="true"
inkscape:snap-text-baseline="true"
inkscape:snap-page="true"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
inkscape:zoom="1.1996734"
inkscape:cx="186.71748"
inkscape:cy="100.02722"
inkscape:window-width="1920"
inkscape:window-height="1007"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="1"
inkscape:current-layer="svg1048" />
<defs
id="defs974">
<linearGradient
id="a">
<stop
offset="0"
stop-color="#fff"
stop-opacity=".098"
id="stop968" />
<stop
offset="1"
stop-color="#fff"
stop-opacity="0"
id="stop970" />
</linearGradient>
<radialGradient
xlink:href="#a"
id="b"
cx="113"
cy="-12.89"
fx="113"
fy="-12.89"
r="59.661999"
gradientTransform="matrix(0,1.96105,-1.97781,0,254.507,78.763)"
gradientUnits="userSpaceOnUse" />
</defs>
<g
transform="translate(-332,-355.362)"
id="g1046">
<rect
style="stroke:none;marker:none"
width="560"
height="164"
x="332"
y="355.362"
rx="20"
ry="20"
color="#000000"
overflow="visible"
stroke="#a6a6a6"
stroke-width="4"
id="rect976" />
<text
y="402.367"
x="508.95099"
style="line-height:100%;-inkscape-font-specification:'DejaVu Sans';marker:none"
color="#000000"
font-weight="400"
font-size="12.395px"
font-family="'DejaVu Sans'"
letter-spacing="0"
word-spacing="0"
overflow="visible"
fill="#ffffff"
id="text980"><tspan
y="402.367"
x="508.95099"
style="-inkscape-font-specification:'DejaVu Sans'"
font-size="34.125px"
id="tspan978">GET IT ON</tspan></text>
<text
style="line-height:100%;-inkscape-font-specification:'Rokkitt Bold';text-align:start;marker:none"
x="508.21301"
y="489.36099"
color="#000000"
font-weight="700"
font-size="29.709px"
font-family="Rokkitt"
letter-spacing="0"
word-spacing="0"
overflow="visible"
fill="#ffffff"
id="text984"><tspan
x="508.21301"
y="489.36099"
style="line-height:100%;-inkscape-font-specification:'Roboto Slab Bold';text-align:start"
font-size="95px"
font-family="'Roboto Slab'"
id="tspan982">F-Droid</tspan></text>
<g
fill-rule="evenodd"
id="g994">
<path
d="m 2.589,1006.862 4.25,5.5"
fill="#8ab000"
stroke="#769616"
stroke-width="2.5"
stroke-linecap="round"
transform="matrix(-2.63159,0,0,2.63157,483.158,-2270.475)"
id="path986" />
<path
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
d="m 476.286,375.862 c 1.193,0.031 2.004,0.497 2.58,1.18 -5.333,6.34 -6.232,7.347 -13.514,16.372 -2.683,3.472 -5.478,1.678 -2.795,-1.793 l 11.185,-14.474 c 0.602,-0.804 1.54,-1.258 2.544,-1.285 z"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#ffffff"
fill-opacity="0.298"
id="path988" />
<path
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
d="m 478.89,377.075 c 0.325,0.39 1.476,2.118 0.058,4.096 l -11.184,14.473 c -2.683,3.471 -3.026,-1.611 -3.026,-1.611 0,0 9.828,-11.869 14.151,-16.958 z"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#263238"
fill-opacity="0.2"
id="path990" />
<path
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
d="m 477.006,376.48 c 1.153,0 2.525,0.373 2.169,2.102 -0.273,1.32 -12.266,15.985 -12.266,15.985 -2.683,3.47 -6.562,1.78 -3.879,-1.691 l 11.143,-14.402 c 0.685,-0.763 1.602,-1.957 2.833,-1.994 z"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#8ab000"
id="path992" />
</g>
<g
fill-rule="evenodd"
id="g1004">
<path
d="m 2.589,1006.862 4.25,5.5"
fill="#8ab000"
stroke="#769616"
stroke-width="2.5"
stroke-linecap="round"
transform="matrix(2.63159,0,0,2.63157,356.842,-2270.475)"
id="path996" />
<path
d="m 363.714,375.862 c -1.193,0.031 -2.004,0.497 -2.58,1.18 5.333,6.34 6.232,7.347 13.514,16.372 2.683,3.472 5.478,1.678 2.795,-1.793 l -11.185,-14.474 c -0.602,-0.804 -1.54,-1.258 -2.544,-1.285 z"
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#ffffff"
fill-opacity="0.298"
id="path998" />
<path
d="m 361.11,377.075 c -0.325,0.39 -1.476,2.118 -0.058,4.096 l 11.184,14.473 c 2.683,3.471 3.026,-1.611 3.026,-1.611 0,0 -9.828,-11.869 -14.151,-16.958 z"
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#263238"
fill-opacity="0.2"
id="path1000" />
<path
d="m 362.995,376.48 c -1.153,0 -2.526,0.373 -2.17,2.102 0.273,1.32 12.266,15.985 12.266,15.985 2.683,3.47 6.562,1.78 3.879,-1.691 l -11.143,-14.402 c -0.685,-0.763 -1.602,-1.957 -2.832,-1.994 z"
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#8ab000"
id="path1002" />
</g>
<g
transform="matrix(2.63159,0,0,2.63157,467.369,-2270.475)"
id="g1014">
<rect
ry="3"
rx="3"
y="1010.36"
x="-37"
height="12.92"
width="38"
fill="#aeea00"
id="rect1006" />
<rect
width="38"
height="10"
x="-37"
y="1013.279"
rx="3"
ry="3"
fill="#263238"
fill-opacity="0.2"
id="rect1008" />
<rect
width="38"
height="10"
x="-37"
y="1010.362"
rx="3"
ry="3"
fill="#ffffff"
fill-opacity="0.298"
id="rect1010" />
<rect
width="38"
height="10.641"
x="-37"
y="1011.5"
rx="3"
ry="2.4560001"
fill="#aeea00"
id="rect1012" />
</g>
<g
transform="matrix(2.63159,0,0,2.63157,356.842,-2270.745)"
id="g1024">
<rect
ry="3"
rx="3"
y="1024.522"
x="5"
height="25.84"
width="38"
fill="#1976d2"
id="rect1016" />
<rect
width="38"
height="13"
x="5"
y="1037.3621"
rx="3"
ry="3"
fill="#263238"
fill-opacity="0.2"
id="rect1018" />
<rect
width="38"
height="13"
x="5"
y="1024.442"
rx="3"
ry="3"
fill="#ffffff"
fill-opacity="0.2"
id="rect1020" />
<rect
width="38"
height="23.559999"
x="5"
y="1025.662"
rx="3"
ry="2.7179999"
fill="#1976d2"
id="rect1022" />
</g>
<g
transform="matrix(2.63159,0,0,2.63157,356.842,396.264)"
id="g1030">
<path
d="m 24,17.75 c -2.88,0 -5.32,1.985 -6.033,4.65 H 21.18 A 3.215,3.215 0 0 1 24,20.75 3.228,3.228 0 0 1 27.25,24 3.228,3.228 0 0 1 24,27.25 3.219,3.219 0 0 1 21.07,25.4 h -3.154 c 0.642,2.766 3.132,4.85 6.084,4.85 3.434,0 6.25,-2.816 6.25,-6.25 0,-3.434 -2.816,-6.25 -6.25,-6.25 z"
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="#0d47a1"
id="path1026" />
<circle
r="9.5500002"
cy="24"
cx="24"
fill="none"
stroke="#0d47a1"
stroke-width="1.9"
stroke-linecap="round"
id="circle1028" />
</g>
<g
transform="matrix(2.63159,0,0,2.63157,356.842,-2269.159)"
id="g1036">
<ellipse
ry="3.875"
rx="3.375"
cx="14.375"
cy="1016.487"
fill="#263238"
fill-opacity="0.2"
id="ellipse1032" />
<circle
r="3.375"
cy="1016.987"
cx="14.375"
fill="#ffffff"
id="circle1034" />
</g>
<g
transform="matrix(2.63159,0,0,2.63157,408.158,-2269.159)"
id="g1042">
<ellipse
cy="1016.487"
cx="14.375"
rx="3.375"
ry="3.875"
fill="#263238"
fill-opacity="0.2"
id="ellipse1038" />
<circle
cx="14.375"
cy="1016.987"
r="3.375"
fill="#ffffff"
id="circle1040" />
</g>
<path
d="m 282.715,299.835 a 3.29,3.29 0 0 0 -2.662,5.336 l 9.474,12.261 A 7.894,7.894 0 0 0 289,320.257 v 18.21 a 7.877,7.877 0 0 0 7.895,7.895 h 84.21 A 7.877,7.877 0 0 0 389,338.468 v -18.211 c 0,-0.999 -0.19,-1.949 -0.525,-2.826 l 9.472,-12.26 a 3.29,3.29 0 0 0 -2.433,-5.334 3.29,3.29 0 0 0 -2.772,1.31 l -9.013,11.666 a 7.91,7.91 0 0 0 -2.624,-0.45 h -84.21 c -0.922,0 -1.8,0.163 -2.622,0.45 l -9.015,-11.666 a 3.29,3.29 0 0 0 -2.543,-1.312 z m 14.18,49.527 A 7.877,7.877 0 0 0 289,357.257 v 52.21 a 7.877,7.877 0 0 0 7.895,7.895 h 84.21 A 7.877,7.877 0 0 0 389,409.468 v -52.211 a 7.877,7.877 0 0 0 -7.895,-7.895 z"
style="line-height:normal;text-indent:0;text-align:start;text-decoration-line:none;text-decoration-style:solid;text-decoration-color:#000000;text-transform:none;isolation:auto;mix-blend-mode:normal;fill:url(#b)"
color="#000000"
font-weight="400"
font-family="sans-serif"
white-space="normal"
overflow="visible"
fill="url(#b)"
fill-rule="evenodd"
transform="translate(81,76)"
id="path1044" />
</g>
</svg>

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

@@ -0,0 +1,39 @@
<svg width="135" height="41" viewBox="0 0 135 41" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M130 40.5H5C2.2 40.5 0 38.3 0 35.5V5.5C0 2.7 2.2 0.5 5 0.5H130C132.8 0.5 135 2.7 135 5.5V35.5C135 38.3 132.8 40.5 130 40.5Z" fill="black"/>
<path d="M47.4 10.7C47.4 11.5 47.2 12.2 46.7 12.7C46.1 13.3 45.4 13.6 44.5 13.6C43.6 13.6 42.9 13.3 42.3 12.7C41.7 12.1 41.4 11.4 41.4 10.5C41.4 9.60002 41.7 8.90002 42.3 8.30002C42.9 7.70002 43.6 7.40002 44.5 7.40002C44.9 7.40002 45.3 7.50002 45.7 7.70002C46.1 7.90002 46.4 8.10002 46.6 8.40002L46.1 8.90002C45.7 8.40002 45.2 8.20002 44.5 8.20002C43.9 8.20002 43.3 8.40002 42.9 8.90002C42.4 9.30002 42.2 9.90002 42.2 10.6C42.2 11.3 42.4 11.9 42.9 12.3C43.4 12.7 43.9 13 44.5 13C45.2 13 45.7 12.8 46.2 12.3C46.5 12 46.7 11.6 46.7 11.1H44.5V10.3H47.4V10.7V10.7ZM52 8.20002H49.3V10.1H51.8V10.8H49.3V12.7H52V13.5H48.5V7.50002H52V8.20002ZM55.3 13.5H54.5V8.20002H52.8V7.50002H57V8.20002H55.3V13.5ZM59.9 13.5V7.50002H60.7V13.5H59.9ZM64.1 13.5H63.3V8.20002H61.6V7.50002H65.7V8.20002H64V13.5H64.1ZM73.6 12.7C73 13.3 72.3 13.6 71.4 13.6C70.5 13.6 69.8 13.3 69.2 12.7C68.6 12.1 68.3 11.4 68.3 10.5C68.3 9.60002 68.6 8.90002 69.2 8.30002C69.8 7.70002 70.5 7.40002 71.4 7.40002C72.3 7.40002 73 7.70002 73.6 8.30002C74.2 8.90002 74.5 9.60002 74.5 10.5C74.5 11.4 74.2 12.1 73.6 12.7ZM69.8 12.2C70.2 12.6 70.8 12.9 71.4 12.9C72 12.9 72.6 12.7 73 12.2C73.4 11.8 73.7 11.2 73.7 10.5C73.7 9.80002 73.5 9.20002 73 8.80002C72.6 8.40002 72 8.10002 71.4 8.10002C70.8 8.10002 70.2 8.30002 69.8 8.80002C69.4 9.20002 69.1 9.80002 69.1 10.5C69.1 11.2 69.3 11.8 69.8 12.2ZM75.6 13.5V7.50002H76.5L79.4 12.2V7.50002H80.2V13.5H79.4L76.3 8.60002V13.5H75.6V13.5Z" fill="white" stroke="white" stroke-width="0.2" stroke-miterlimit="10"/>
<path d="M68.1 22.3C65.7 22.3 63.8 24.1 63.8 26.6C63.8 29 65.7 30.9 68.1 30.9C70.5 30.9 72.4 29.1 72.4 26.6C72.4 24 70.5 22.3 68.1 22.3ZM68.1 29.1C66.8 29.1 65.7 28 65.7 26.5C65.7 25 66.8 23.9 68.1 23.9C69.4 23.9 70.5 24.9 70.5 26.5C70.5 28 69.4 29.1 68.1 29.1ZM58.8 22.3C56.4 22.3 54.5 24.1 54.5 26.6C54.5 29 56.4 30.9 58.8 30.9C61.2 30.9 63.1 29.1 63.1 26.6C63.1 24 61.2 22.3 58.8 22.3ZM58.8 29.1C57.5 29.1 56.4 28 56.4 26.5C56.4 25 57.5 23.9 58.8 23.9C60.1 23.9 61.2 24.9 61.2 26.5C61.2 28 60.1 29.1 58.8 29.1ZM47.7 23.6V25.4H52C51.9 26.4 51.5 27.2 51 27.7C50.4 28.3 49.4 29 47.7 29C45 29 43 26.9 43 24.2C43 21.5 45.1 19.4 47.7 19.4C49.1 19.4 50.2 20 51 20.7L52.3 19.4C51.2 18.4 49.8 17.6 47.8 17.6C44.2 17.6 41.1 20.6 41.1 24.2C41.1 27.8 44.2 30.8 47.8 30.8C49.8 30.8 51.2 30.2 52.4 28.9C53.6 27.7 54 26 54 24.7C54 24.3 54 23.9 53.9 23.6H47.7V23.6ZM93.1 25C92.7 24 91.7 22.3 89.5 22.3C87.3 22.3 85.5 24 85.5 26.6C85.5 29 87.3 30.9 89.7 30.9C91.6 30.9 92.8 29.7 93.2 29L91.8 28C91.3 28.7 90.7 29.2 89.7 29.2C88.7 29.2 88.1 28.8 87.6 27.9L93.3 25.5L93.1 25V25ZM87.3 26.4C87.3 24.8 88.6 23.9 89.5 23.9C90.2 23.9 90.9 24.3 91.1 24.8L87.3 26.4ZM82.6 30.5H84.5V18H82.6V30.5ZM79.6 23.2C79.1 22.7 78.3 22.2 77.3 22.2C75.2 22.2 73.2 24.1 73.2 26.5C73.2 28.9 75.1 30.7 77.3 30.7C78.3 30.7 79.1 30.2 79.5 29.7H79.6V30.3C79.6 31.9 78.7 32.8 77.3 32.8C76.2 32.8 75.4 32 75.2 31.3L73.6 32C74.1 33.1 75.3 34.5 77.4 34.5C79.6 34.5 81.4 33.2 81.4 30.1V22.5H79.6V23.2V23.2ZM77.4 29.1C76.1 29.1 75 28 75 26.5C75 25 76.1 23.9 77.4 23.9C78.7 23.9 79.7 25 79.7 26.5C79.7 28 78.7 29.1 77.4 29.1ZM101.8 18H97.3V30.5H99.2V25.8H101.8C103.9 25.8 105.9 24.3 105.9 21.9C105.9 19.5 103.9 18 101.8 18V18ZM101.9 24H99.2V19.7H101.9C103.3 19.7 104.1 20.9 104.1 21.8C104 22.9 103.2 24 101.9 24ZM113.4 22.2C112 22.2 110.6 22.8 110.1 24.1L111.8 24.8C112.2 24.1 112.8 23.9 113.5 23.9C114.5 23.9 115.4 24.5 115.5 25.5V25.6C115.2 25.4 114.4 25.1 113.6 25.1C111.8 25.1 110 26.1 110 27.9C110 29.6 111.5 30.7 113.1 30.7C114.4 30.7 115 30.1 115.5 29.5H115.6V30.5H117.4V25.7C117.2 23.5 115.5 22.2 113.4 22.2V22.2ZM113.2 29.1C112.6 29.1 111.7 28.8 111.7 28C111.7 27 112.8 26.7 113.7 26.7C114.5 26.7 114.9 26.9 115.4 27.1C115.2 28.3 114.2 29.1 113.2 29.1V29.1ZM123.7 22.5L121.6 27.9H121.5L119.3 22.5H117.3L120.6 30.1L118.7 34.3H120.6L125.7 22.5H123.7V22.5ZM106.9 30.5H108.8V18H106.9V30.5Z" fill="white"/>
<path d="M10.4 8C10.1 8.3 10 8.8 10 9.4V31.5C10 32.1 10.2 32.6 10.5 32.9L10.6 33L23 20.6V20.4L10.4 8Z" fill="url(#paint0_linear_7_408)"/>
<path d="M27 24.8L22.9 20.7V20.4L27 16.3L27.1 16.4L32 19.2C33.4 20 33.4 21.3 32 22.1L27 24.8V24.8Z" fill="url(#paint1_linear_7_408)"/>
<path d="M27.1 24.7L22.9 20.5L10.4 33C10.9 33.5 11.6 33.5 12.5 33.1L27.1 24.7" fill="url(#paint2_linear_7_408)"/>
<path d="M27.1 16.3001L12.5 8.00005C11.6 7.50005 10.9 7.60005 10.4 8.10005L22.9 20.5001L27.1 16.3001V16.3001Z" fill="url(#paint3_linear_7_408)"/>
<path opacity="0.2" d="M27 24.6L12.5 32.8C11.7 33.3 11 33.2 10.5 32.8L10.4 32.9L10.5 33C11 33.4 11.7 33.5 12.5 33L27 24.6Z" fill="black"/>
<path opacity="0.12" d="M10.4 32.8C10.1 32.5 10 32 10 31.4V31.5C10 32.1 10.2 32.6 10.5 32.9V32.8H10.4ZM32 21.8L27 24.6L27.1 24.7L32 21.9C32.7 21.5 33 21 33 20.5C33 21 32.6 21.4 32 21.8V21.8Z" fill="black"/>
<path opacity="0.25" d="M12.5 8.10003L32 19.2C32.6 19.6 33 20 33 20.5C33 20 32.7 19.5 32 19.1L12.5 8.00003C11.1 7.20003 10 7.80003 10 9.40003V9.50003C10 8.00003 11.1 7.30003 12.5 8.10003Z" fill="white"/>
<defs>
<linearGradient id="paint0_linear_7_408" x1="21.8" y1="9.21" x2="5.017" y2="25.992" gradientUnits="userSpaceOnUse">
<stop stop-color="#00A0FF"/>
<stop offset="0.007" stop-color="#00A1FF"/>
<stop offset="0.26" stop-color="#00BEFF"/>
<stop offset="0.512" stop-color="#00D2FF"/>
<stop offset="0.76" stop-color="#00DFFF"/>
<stop offset="1" stop-color="#00E3FF"/>
</linearGradient>
<linearGradient id="paint1_linear_7_408" x1="33.834" y1="20.501" x2="9.63699" y2="20.501" gradientUnits="userSpaceOnUse">
<stop stop-color="#FFE000"/>
<stop offset="0.409" stop-color="#FFBD00"/>
<stop offset="0.775" stop-color="#FFA500"/>
<stop offset="1" stop-color="#FF9C00"/>
</linearGradient>
<linearGradient id="paint2_linear_7_408" x1="24.827" y1="22.796" x2="2.069" y2="45.554" gradientUnits="userSpaceOnUse">
<stop stop-color="#FF3A44"/>
<stop offset="1" stop-color="#C31162"/>
</linearGradient>
<linearGradient id="paint3_linear_7_408" x1="7.29699" y1="0.676051" x2="17.46" y2="10.8391" gradientUnits="userSpaceOnUse">
<stop stop-color="#32A071"/>
<stop offset="0.069" stop-color="#2DA771"/>
<stop offset="0.476" stop-color="#15CF74"/>
<stop offset="0.801" stop-color="#06E775"/>
<stop offset="1" stop-color="#00F076"/>
</linearGradient>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 6.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.1 KiB

@@ -0,0 +1,10 @@
<svg width="34" height="35" viewBox="0 0 34 35" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M3.02958 8.60922L8.622 14.2013L14.3705 8.45375L17.1669 11.2498L11.4183 16.9972L17.0114 22.5895L14.1373 25.4633L8.54422 19.871L2.79636 25.6187L0 22.8227L5.74794 17.075L0.155484 11.483L3.02958 8.60922Z" fill="white"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M14.0923 25.5156L16.944 22.6642L16.9429 22.6634L22.6467 16.9612L17.0513 11.3675L17.0523 11.367L14.2548 8.56979L8.65972 2.97535L11.5114 0.123963L17.1061 5.71849L22.8099 0.015625L25.6074 2.81285L19.9035 8.51562L25.4984 14.1099L31.2025 8.40729L34 11.2045L28.2958 16.907L33.8917 22.5017L31.0399 25.3531L25.4442 19.7584L19.7409 25.4611L25.3365 31.0559L22.4848 33.9073L16.8892 28.3124L11.1864 34.0156L8.38885 31.2184L14.0923 25.5156Z" fill="url(#paint0_linear_656_10815)"/>
<defs>
<linearGradient id="paint0_linear_656_10815" x1="12.8381" y1="-0.678252" x2="9.54355" y2="31.4493" gradientUnits="userSpaceOnUse">
<stop stop-color="#01F1FF"/>
<stop offset="1" stop-color="#0197FF"/>
</linearGradient>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 1.1 KiB

@@ -0,0 +1,15 @@
<svg width="34" height="34" viewBox="0 0 34 34" fill="none" xmlns="http://www.w3.org/2000/svg">
<g clip-path="url(#clip0_14_10)">
<path fill-rule="evenodd" clip-rule="evenodd" d="M3.02972 8.59396L8.62219 14.186L14.3703 8.43848L17.1668 11.2346L11.4182 16.982L17.0112 22.5742L14.1371 25.448L8.5441 19.8557L2.79651 25.6035L0 22.8074L5.74813 17.0597L0.155656 11.4678L3.02972 8.59396Z" fill="#023789"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M14.0922 25.5L16.9434 22.6486L16.9423 22.6478L22.6464 16.9456L17.0512 11.3519L17.0518 11.3514L14.2542 8.55418L8.65961 2.95973L11.5114 0.108337L17.106 5.70288L22.8095 0L25.607 2.79722L19.903 8.5L25.4981 14.0943L31.2022 8.39169L33.9997 11.1889L28.2957 16.8914L33.8914 22.4861L31.0396 25.3375L25.4439 19.7428L19.7404 25.4454L25.3361 31.0403L22.4843 33.8917L16.8887 28.2968L11.1862 34L8.38867 31.2028L14.0922 25.5Z" fill="url(#paint0_linear_14_10)"/>
</g>
<defs>
<linearGradient id="paint0_linear_14_10" x1="12.8379" y1="-0.693875" x2="9.54344" y2="31.4337" gradientUnits="userSpaceOnUse">
<stop stop-color="#01F1FF"/>
<stop offset="1" stop-color="#0197FF"/>
</linearGradient>
<clipPath id="clip0_14_10">
<rect width="34" height="34" fill="white"/>
</clipPath>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

+3
View File
@@ -0,0 +1,3 @@
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M2.92814 12.6789C6.17655 15.9347 11.7044 15.6463 14.5425 12.0117C14.6636 11.8566 14.6825 11.6448 14.5907 11.4708C14.4989 11.2967 14.3136 11.1926 14.1172 11.2049C11.5269 11.3673 8.97627 10.4315 7.0743 8.52765C5.17264 6.62414 4.23958 4.06868 4.40169 1.47281C4.41397 1.2762 4.30965 1.09069 4.13526 0.999048C3.96088 0.907402 3.74893 0.926696 3.59396 1.04833C3.36099 1.23117 3.13828 1.42685 2.92823 1.63726C-0.111372 4.68223 -0.111585 9.63533 2.92814 12.6789Z" stroke="black" stroke-miterlimit="10" stroke-linejoin="round"/>
</svg>

After

Width:  |  Height:  |  Size: 632 B

File diff suppressed because one or more lines are too long
+39
View File
@@ -0,0 +1,39 @@
const isMobile = {
Android: () => navigator.userAgent.match(/Android/i),
iOS: () => navigator.userAgent.match(/iPhone|iPad|iPod/i)
};
window.addEventListener('click', clickHandler)
if (isMobile.iOS) {
for (const btn of document.getElementsByClassName("close-overlay-btn")) {
btn.addEventListener("touchend", (e) => setTimeout(() => closeOverlay(e), 100))
}
}
function clickHandler(e) {
if (e.target.closest('.contact-tab-btn')) {
e.target.closest('.contact-tab').classList.toggle('active')
}
}
window.addEventListener('load', () => {
const googlePlayBtn = document.querySelector('.google-play-btn');
const appleStoreBtn = document.querySelector('.apple-store-btn');
const fDroidBtn = document.querySelector('.f-droid-btn');
if (!googlePlayBtn || !appleStoreBtn || !fDroidBtn) return;
if (isMobile.Android()) {
googlePlayBtn.classList.remove('hidden');
fDroidBtn.classList.remove('hidden');
}
else if (isMobile.iOS()) {
appleStoreBtn.classList.remove('hidden');
}
else {
appleStoreBtn.classList.remove('hidden');
googlePlayBtn.classList.remove('hidden');
fDroidBtn.classList.remove('hidden');
}
})
+414
View File
@@ -0,0 +1,414 @@
@font-face {
font-family: Gilroy;
src: url("GilroyRegular.woff2") format("woff2");
font-weight: 400;
font-style: normal;
}
@font-face {
font-family: Gilroy;
src: url("GilroyLight.woff2") format("woff2");
font-weight: 300;
font-style: normal;
}
@font-face {
font-family: Gilroy;
src: url("GilroyMedium.woff2") format("woff2");
font-weight: 500;
font-style: normal;
}
@font-face {
font-family: Gilroy;
src: url("GilroyBold.woff2") format("woff2");
font-weight: 700;
font-style: normal;
}
@font-face {
font-family: Gilroy;
src: url("GilroyRegularItalic.woff2") format("woff2");
font-weight: 400;
font-style: italic;
}
html {
scroll-behavior: smooth;
font-family: Gilroy, Helvetica, sans-serif;
;
letter-spacing: 0.003em;
}
img {
user-select: none;
-webkit-user-select: none;
/* For Safari and older Chrome versions */
-moz-user-select: none;
/* For Firefox */
-ms-user-select: none;
/* For Internet Explorer and Edge */
}
a{
word-wrap: break-word;
}
/* NEW SITE */
.container,
.container-fluid,
.container-xxl,
.container-xl,
.container-lg,
.container-md,
.container-sm {
width: 100%;
/* padding: 0 20px; */
margin-right: auto;
margin-left: auto;
}
@media (min-width: 576px) {
.container-sm,
.container {
max-width: 540px;
}
}
@media (min-width: 768px) {
.container-md,
.container-sm,
.container {
max-width: 720px;
}
}
@media (min-width: 992px) {
.container-lg,
.container-md,
.container-sm,
.container {
max-width: 960px;
}
}
@media (min-width: 1200px) {
.container-xl,
.container-lg,
.container-md,
.container-sm,
.container {
max-width: 1140px;
}
}
@media (min-width: 1400px) {
.container-xxl,
.container-xl,
.container-lg,
.container-md,
.container-sm,
.container {
max-width: 1320px;
}
}
.gradient-text {
background: -webkit-linear-gradient(to bottom, #53C1FF -50%, #0053D0 160%);
background: linear-gradient(to bottom, #53C1FF -50%, #0053D0 160%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-fill-color: transparent;
}
.dark .border-gradient {
background:
linear-gradient(#11182F, #11182F) padding-box,
linear-gradient(to bottom, transparent, #01F1FF 58%) border-box;
border: 1px solid transparent;
}
.dark .only-light {
display: none;
}
.only-dark {
display: none;
}
.dark .only-dark {
display: inherit;
}
.menu-link {
font-size: 16px;
line-height: 33.42px;
color: #0D0E12;
}
.dark .menu-link {
color: #fff;
}
.nav-link ul li a.active {
color: #0053D0;
}
.dark .nav-link ul li a.active {
color: #66D9E2;
}
@media (min-width:1024px) {
.nav-link-text,
.menu-link {
display: inline-block;
position: relative;
color: #0D0E12;
}
.nav-link-text::before,
.active .nav-link-text::before,
.menu-link::before {
content: "";
position: absolute;
width: 0;
height: 1px;
bottom: 0;
right: 0;
/* background-color: initial; */
transition: width 0.25s ease-out;
}
.menu-link::before {
background-color: #0D0E12;
}
.dark .menu-link::before {
background-color: #fff;
}
.active .nav-link-text::before {
width: 100%;
}
.nav-link:hover .nav-link-text::before,
.menu-link:hover::before {
width: 100%;
left: 0;
right: auto;
}
}
.sub-menu {
visibility: hidden;
opacity: 0;
color: #505158;
}
.sub-menu .no-hover {
color: #505158 !important;
}
.dark .sub-menu,
.dark .sub-menu .no-hover {
color: #fff !important;
}
.dark .sub-menu li:hover {
color: #66D9E2;
}
.sub-menu li:hover {
color: #0053D0;
}
.sub-menu {
transition: all .3s ease !important;
}
.nav-link span svg,
header nav {
transition: all 0.5s ease;
}
.nav-link:hover span svg {
transform: rotate(180deg);
}
@media (min-width:1024px) {
.nav-link:hover .sub-menu,
.nav-link:focus-within .sub-menu {
visibility: visible;
opacity: 1;
margin-top: 0;
}
}
@media (max-width: 1024px) {
.sub-menu {
max-height: 0;
transform: translateY(-10px);
transition: all .7s ease !important;
}
.active .sub-menu {
max-height: 600px;
transform: translateY(0px);
opacity: 1;
visibility: visible;
margin-top: 0;
}
header nav {
visibility: hidden;
opacity: 0;
transform: translateX(100%);
}
header nav.open {
visibility: visible;
opacity: 1;
transform: translateX(0);
}
}
.lock-scroll {
overflow: hidden;
}
/* hero */
header {
transition: all .7s ease;
}
.primary-header {
background: linear-gradient(270deg, #0053D0 35.85%, #0197FF 94.78%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-shadow: 0px 4px 74px #e9e7e2;
}
.dark .primary-header {
background: linear-gradient(270deg, #70F0F9 100%, #70F0F9 100%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-shadow: none;
}
.secondary-header {
color: #606c71;
text-shadow: 0px 4px 74px #e9e7e2;
}
.dark .secondary-header {
color: #fff;
text-shadow: none;
}
.description {
width: 31rem;
}
p a {
color: #0053D0;
text-decoration: underline;
text-underline-offset: 2px;
}
.dark p a {
color: #70F0F9;
}
/* For Contact & Invitation Page */
.primary-header-contact {
background: linear-gradient(251.16deg, #53c1ff 1.1%, #0053d0 100.82%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-shadow: 0px 4px 74px #e9e7e2;
}
.dark .primary-header-contact {
background: linear-gradient(270deg, #70F0F9 100%, #70F0F9 100%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-shadow: none;
}
.secondary-header-contact {
text-shadow: 0px 4px 74px #e9e7e2;
}
.dark .secondary-header-contact {
text-shadow: none;
}
.content_copy_with_tooltip {
background-color: #f8f8f6;
border-radius: 50px;
padding-bottom: 4px;
padding-top: 8px;
margin-top: 16px;
margin-bottom: 16px;
}
.content_copy_with_tooltip .tooltip {
vertical-align: -6px;
}
.content_copy_with_tooltip .content {
font-size: 15px;
}
.contact-tab>.contact-tab-content,
.job-tab>.job-tab-content {
opacity: 0;
max-height: 0;
transition: all 0.5s ease;
visibility: hidden;
transform: translateY(10px);
overflow: hidden;
}
.contact-tab svg,
.job-tab svg {
transform: rotate(-180deg);
transition: all .5s ease;
}
.contact-tab.active>.contact-tab-content,
.job-tab.active>.job-tab-content {
opacity: 1;
max-height: 300px;
visibility: visible;
transform: translateY(0px);
}
.for-tablet .contact-tab.active>.contact-tab-content,
.for-tablet .job-tab.active>.job-tab-content {
min-height: 450px;
}
.contact-tab.active svg,
.contact-tab:hover svg,
.job-tab.active svg,
.job-tab:hover svg {
transform: rotate(0deg);
}
.d-none-if-js-disabled {
display: none !important;
}
+11
View File
@@ -0,0 +1,11 @@
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M23.9958 12.2815C23.9363 12.3954 23.8849 12.5146 23.8158 12.6224C23.5945 12.9674 23.2766 13.1624 22.8654 13.1685C22.2182 13.1782 21.5707 13.1764 20.9234 13.17C20.2781 13.1636 19.7772 12.6464 19.7773 11.9999C19.7774 11.3535 20.2754 10.8389 20.9237 10.8303C21.532 10.8221 22.1405 10.8258 22.7488 10.8278C23.3522 10.8298 23.7281 11.0908 23.9462 11.6508C23.956 11.6761 23.9789 11.6964 23.9958 11.719C23.9958 11.9064 23.9958 12.094 23.9958 12.2815Z" fill="white"/>
<path d="M11.7154 24.0003C11.6217 23.9526 11.5256 23.9088 11.4345 23.8564C11.0545 23.6377 10.836 23.3104 10.8286 22.87C10.8175 22.2149 10.8179 21.5593 10.828 20.9042C10.8378 20.2738 11.3597 19.7812 11.9967 19.7812C12.6336 19.7812 13.155 20.2739 13.1654 20.9042C13.1757 21.5359 13.1717 22.168 13.1682 22.7998C13.1652 23.3392 12.8885 23.7369 12.3906 23.937C12.3509 23.9529 12.3153 23.979 12.2779 24.0003C12.0904 24.0003 11.9029 24.0003 11.7154 24.0003Z" fill="white"/>
<path d="M17.2592 11.9958C17.2733 14.8825 14.9232 17.2468 12.0032 17.2612C9.11732 17.2754 6.75397 14.9264 6.73836 12.0041C6.72295 9.12027 9.07502 6.75326 11.9946 6.73835C14.8788 6.72363 17.2449 9.07587 17.2592 11.9958Z" stroke="white" stroke-width="1.5"/>
<path d="M13.1693 2.11324C13.1692 2.43329 13.1744 2.75345 13.1682 3.07341C13.1555 3.7216 12.6425 4.21934 11.995 4.21864C11.3493 4.21789 10.8358 3.71808 10.828 3.06768C10.8204 2.42766 10.8201 1.78736 10.8283 1.14738C10.8365 0.500704 11.3562 -0.000843934 12.0007 1.06615e-06C12.6437 0.000846066 13.1564 0.504314 13.1684 1.15307C13.1743 1.47303 13.1694 1.79318 13.1693 2.11324Z" fill="white"/>
<path d="M2.10878 13.1714C1.78877 13.1714 1.46872 13.1754 1.14885 13.1705C0.504832 13.1605 -0.000422735 12.6426 2.65407e-07 11.9987C0.000423265 11.3553 0.503376 10.838 1.15138 10.8301C1.79126 10.8223 2.43138 10.8222 3.07126 10.8303C3.72034 10.8385 4.21822 11.3541 4.21794 12.0012C4.21766 12.6477 3.71555 13.1609 3.06872 13.1706C2.7488 13.1753 2.42875 13.1714 2.10878 13.1714Z" fill="white"/>
<path d="M6.85268 5.524C6.82732 6.152 6.60944 6.52005 6.16969 6.72981C5.73844 6.93552 5.29738 6.90378 4.94534 6.58208C4.41604 6.09838 3.90431 5.59148 3.42451 5.05894C3.02923 4.62023 3.09727 3.9209 3.51626 3.50792C3.9361 3.09409 4.63284 3.03567 5.06893 3.43194C5.59381 3.90888 6.09569 4.41446 6.57188 4.93996C6.73726 5.12252 6.79642 5.4014 6.85268 5.524Z" fill="white"/>
<path d="M17.1426 18.4446C17.1749 17.8424 17.389 17.4819 17.8198 17.2738C18.2418 17.07 18.6812 17.0888 19.0265 17.3998C19.5706 17.8899 20.0919 18.4099 20.5814 18.9544C20.9675 19.384 20.895 20.0764 20.485 20.4873C20.0824 20.8907 19.3961 20.9756 18.9718 20.5988C18.4129 20.1026 17.89 19.5625 17.3864 19.0096C17.2323 18.8405 17.1926 18.5671 17.1426 18.4446Z" fill="white"/>
<path d="M18.2026 6.84235C17.8449 6.82333 17.4821 6.61377 17.2723 6.18256C17.0626 5.7515 17.0878 5.30837 17.4061 4.95629C17.8919 4.41887 18.4055 3.90234 18.945 3.41897C19.3826 3.02693 20.0905 3.1037 20.4947 3.52429C20.9057 3.95198 20.9561 4.64003 20.5568 5.07805C20.0843 5.59645 19.576 6.08306 19.071 6.5708C18.8688 6.76619 18.6057 6.84832 18.2026 6.84235Z" fill="white"/>
<path d="M5.54205 17.1445C6.14812 17.1747 6.51058 17.385 6.72137 17.8153C6.9323 18.2459 6.90765 18.6892 6.58942 19.0415C6.1037 19.5791 5.58933 20.0948 5.05088 20.5795C4.62165 20.9659 3.93035 20.8989 3.51681 20.4933C3.09875 20.0833 3.02864 19.3789 3.4227 18.942C3.908 18.404 4.42706 17.8934 4.96382 17.4066C5.13982 17.247 5.41663 17.1985 5.54205 17.1445Z" fill="white"/>
</svg>

After

Width:  |  Height:  |  Size: 3.5 KiB

File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

-1
View File
@@ -15,7 +15,6 @@ logCfg = LogConfig {lc_file = Nothing, lc_stderr = True}
main :: IO ()
main = do
setLogLevel LogDebug -- change to LogError in production
cfgPath <- getEnvPath "NTF_SERVER_CFG_PATH" defaultCfgPath
logPath <- getEnvPath "NTF_SERVER_LOG_PATH" defaultLogPath
withGlobalLogging logCfg $ ntfServerCLI cfgPath logPath
+3 -1
View File
@@ -1,4 +1,5 @@
{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TypeApplications #-}
@@ -38,7 +39,8 @@ logCfg = LogConfig {lc_file = Nothing, lc_stderr = True}
-- Warning: this SMP agent server is experimental - it does not work correctly with multiple connected TCP clients in some cases.
main :: IO ()
main = do
putStrLn $ "SMP agent listening on port " ++ tcpPort (cfg :: AgentConfig)
let AgentConfig {tcpPort} = cfg
putStrLn $ maybe (error "no agent port") (\port -> "SMP agent listening on port " ++ port) tcpPort
setLogLevel LogInfo -- LogError
Right st <- createAgentStore agentDbFile agentDbKey False MCConsole
withGlobalLogging logCfg $ runSMPAgent (transport @TLS) cfg servers st
+4 -5
View File
@@ -1,10 +1,10 @@
{-# LANGUAGE LambdaCase #-}
module Main where
import Control.Logger.Simple
import Simplex.Messaging.Server.CLI (getEnvPath)
import Simplex.Messaging.Server.Main
import Simplex.Messaging.Server.Main (smpServerCLI_)
import Simplex.Messaging.Server.Web (serveStaticFiles, attachStaticFiles)
import SMPWeb (smpGenerateSite)
defaultCfgPath :: FilePath
defaultCfgPath = "/etc/opt/simplex"
@@ -17,7 +17,6 @@ logCfg = LogConfig {lc_file = Nothing, lc_stderr = True}
main :: IO ()
main = do
setLogLevel LogDebug
cfgPath <- getEnvPath "SMP_SERVER_CFG_PATH" defaultCfgPath
logPath <- getEnvPath "SMP_SERVER_LOG_PATH" defaultLogPath
withGlobalLogging logCfg $ smpServerCLI cfgPath logPath
withGlobalLogging logCfg $ smpServerCLI_ smpGenerateSite serveStaticFiles attachStaticFiles cfgPath logPath
+43
View File
@@ -0,0 +1,43 @@
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
module SMPWeb
( smpGenerateSite,
serverInformation,
) where
import Data.ByteString (ByteString)
import Data.String (fromString)
import Web.Embedded (embeddedContent)
import Simplex.Messaging.Encoding.String (strEncode)
import Simplex.Messaging.Server.Information
import Simplex.Messaging.Server.Main (simplexmqSource)
import qualified Simplex.Messaging.Server.Web as Web
import Simplex.Messaging.Server.Web (render, serverInfoSubsts, timedTTLText)
import Simplex.Messaging.Transport.Client (TransportHost (..))
smpGenerateSite :: ServerInformation -> Maybe TransportHost -> FilePath -> IO ()
smpGenerateSite si onionHost path =
Web.generateSite embeddedContent (serverInformation si onionHost) smpLinkPages path
smpLinkPages :: [String]
smpLinkPages = ["contact", "invitation", "a", "c", "g", "r", "i"]
serverInformation :: ServerInformation -> Maybe TransportHost -> ByteString
serverInformation ServerInformation {config, information} onionHost = render (Web.indexHtml embeddedContent) substs
where
substs = [("smpConfig", Just "y"), ("xftpConfig", Nothing)] <> substConfig <> serverInfoSubsts simplexmqSource information <> [("onionHost", strEncode <$> onionHost), ("iniFileName", Just "smp-server.ini")]
substConfig =
[ ( "persistence",
Just $ case persistence config of
SPMMemoryOnly -> "In-memory only"
SPMQueues -> "Queues"
SPMMessages -> "Queues and messages"
),
("messageExpiration", Just $ maybe "Never" (fromString . timedTTLText) $ messageExpiration config),
("statsEnabled", Just . yesNo $ statsEnabled config),
("newQueuesAllowed", Just . yesNo $ newQueuesAllowed config),
("basicAuthEnabled", Just . yesNo $ basicAuthEnabled config)
]
yesNo True = "Yes"
yesNo False = "No"
+4 -2
View File
@@ -1,8 +1,10 @@
module Main where
import Control.Logger.Simple
import Simplex.FileTransfer.Server.Main (xftpServerCLI_)
import Simplex.Messaging.Server.CLI (getEnvPath)
import Simplex.FileTransfer.Server.Main
import Simplex.Messaging.Server.Web (serveStaticFiles)
import XFTPWeb (xftpGenerateSite)
defaultCfgPath :: FilePath
defaultCfgPath = "/etc/opt/simplex-xftp"
@@ -18,4 +20,4 @@ main = do
setLogLevel LogDebug -- change to LogError in production
cfgPath <- getEnvPath "XFTP_SERVER_CFG_PATH" defaultCfgPath
logPath <- getEnvPath "XFTP_SERVER_LOG_PATH" defaultLogPath
withGlobalLogging logCfg $ xftpServerCLI cfgPath logPath
withGlobalLogging logCfg $ xftpServerCLI_ xftpGenerateSite serveStaticFiles cfgPath logPath
+67
View File
@@ -0,0 +1,67 @@
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TemplateHaskell #-}
module XFTPWeb
( xftpGenerateSite,
xftpServerInformation,
) where
import Control.Monad (forM_)
import qualified Data.ByteString.Char8 as B
import Data.ByteString (ByteString)
import Data.FileEmbed (embedDir, embedFile)
import Data.Maybe (isJust)
import Data.String (fromString)
import Web.Embedded (embeddedContent)
import Simplex.FileTransfer.Server.Env (XFTPServerConfig (..))
import Simplex.Messaging.Encoding.String (strEncode)
import Simplex.Messaging.Server.Expiration (ExpirationConfig (..))
import Simplex.Messaging.Server.Information (ServerPublicInfo)
import Simplex.Messaging.Server.Main (simplexmqSource)
import qualified Simplex.Messaging.Server.Web as Web
import Simplex.Messaging.Server.Web (render, serverInfoSubsts, timedTTLText)
import Simplex.Messaging.Transport.Client (TransportHost (..))
import System.Directory (createDirectoryIfMissing)
import System.FilePath ((</>))
xftpWebContent :: [(FilePath, ByteString)]
xftpWebContent = $(embedDir "apps/xftp-server/static/xftp-web-bundle/")
xftpMediaContent :: [(FilePath, ByteString)]
xftpMediaContent = $(embedDir "apps/xftp-server/static/media/")
xftpFilePageHtml :: ByteString
xftpFilePageHtml = $(embedFile "apps/xftp-server/static/file.html")
xftpGenerateSite :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> FilePath -> IO ()
xftpGenerateSite cfg info onionHost path = do
let substs = xftpSubsts cfg info onionHost
Web.generateSite embeddedContent (render (Web.indexHtml embeddedContent) substs) [] path
let xftpDir = path </> "xftp-web-bundle"
mediaDir = path </> "media"
fileDir = path </> "file"
filePage xftpDir xftpWebContent
filePage mediaDir xftpMediaContent
createDirectoryIfMissing True fileDir
B.writeFile (fileDir </> "index.html") $ render xftpFilePageHtml substs
where
filePage dir content_ = do
createDirectoryIfMissing True dir
forM_ content_ $ \(fp, content) -> B.writeFile (dir </> fp) content
xftpServerInformation :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> ByteString
xftpServerInformation cfg info onionHost = render (Web.indexHtml embeddedContent) (xftpSubsts cfg info onionHost)
xftpSubsts :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> [(ByteString, Maybe ByteString)]
xftpSubsts XFTPServerConfig {fileExpiration, logStatsInterval, allowNewFiles, newFileBasicAuth} information onionHost =
[("smpConfig", Nothing), ("xftpConfig", Just "y")] <> substConfig <> serverInfoSubsts simplexmqSource information <> [("onionHost", strEncode <$> onionHost), ("iniFileName", Just "file-server.ini")]
where
substConfig =
[ ("fileExpiration", Just $ maybe "Never" (fromString . timedTTLText . ttl) fileExpiration),
("statsEnabled", Just . yesNo $ isJust logStatsInterval),
("newUploadsAllowed", Just . yesNo $ allowNewFiles),
("basicAuthEnabled", Just . yesNo $ isJust newFileBasicAuth)
]
yesNo True = "Yes"
yesNo False = "No"
File diff suppressed because one or more lines are too long
@@ -0,0 +1,115 @@
<svg width="440" height="520" viewBox="-20 0 440 520" fill="none" xmlns="http://www.w3.org/2000/svg">
<!-- Sender browser -->
<rect x="120" y="16" width="160" height="56" rx="10" stroke="#70F0F9" stroke-width="1.5"/>
<text x="200" y="40" text-anchor="middle" font-family="system-ui, sans-serif" font-size="13" font-weight="600" fill="#70F0F9">Sender's browser</text>
<text x="200" y="56" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" fill="rgba(112,240,249,0.7)">encrypts file</text>
<!-- Arrow down from sender to chunks -->
<line x1="200" y1="72" x2="200" y2="120" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<!-- Chunks row -->
<rect x="112" y="120" width="176" height="40" rx="8" fill="none" stroke="#70F0F9" stroke-width="1" stroke-dasharray="4 3"/>
<text x="200" y="145" text-anchor="middle" font-family="system-ui, sans-serif" font-size="12" fill="#70F0F9">encrypted chunks</text>
<!-- Arrows from chunks to routers -->
<line x1="152" y1="160" x2="80" y2="220" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<line x1="200" y1="160" x2="200" y2="220" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<line x1="248" y1="160" x2="320" y2="220" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<!-- Router 1 (SimpleX) -->
<rect x="20" y="220" width="120" height="56" rx="6" fill="none" stroke="#70F0F9" stroke-width="1.5"/>
<g transform="translate(28, 227)">
<rect width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="6" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="12" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<circle cx="11" cy="2" r="1" fill="#70F0F9"/>
<circle cx="11" cy="8" r="1" fill="#70F0F9"/>
<circle cx="11" cy="14" r="1" fill="#70F0F9"/>
</g>
<text x="80" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#70F0F9">SimpleX</text>
<text x="80" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="rgba(112,240,249,0.7)">XFTP router</text>
<!-- Router 2 (Flux) -->
<rect x="155" y="220" width="90" height="56" rx="6" fill="none" stroke="#70F0F9" stroke-width="1.5"/>
<g transform="translate(163, 227)">
<rect width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="6" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="12" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<circle cx="11" cy="2" r="1" fill="#70F0F9"/>
<circle cx="11" cy="8" r="1" fill="#70F0F9"/>
<circle cx="11" cy="14" r="1" fill="#70F0F9"/>
</g>
<text x="200" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#70F0F9">Flux</text>
<text x="200" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="rgba(112,240,249,0.7)">XFTP router</text>
<!-- Router 3 (SimpleX) -->
<rect x="260" y="220" width="120" height="56" rx="6" fill="none" stroke="#70F0F9" stroke-width="1.5"/>
<g transform="translate(268, 227)">
<rect width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="6" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<rect y="12" width="14" height="4" rx="1" fill="rgba(112,240,249,0.5)"/>
<circle cx="11" cy="2" r="1" fill="#70F0F9"/>
<circle cx="11" cy="8" r="1" fill="#70F0F9"/>
<circle cx="11" cy="14" r="1" fill="#70F0F9"/>
</g>
<text x="320" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#70F0F9">SimpleX</text>
<text x="320" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="rgba(112,240,249,0.7)">XFTP router</text>
<!-- Arrows from routers down -->
<line x1="80" y1="276" x2="152" y2="336" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<line x1="200" y1="276" x2="200" y2="336" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<line x1="320" y1="276" x2="248" y2="336" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<!-- Re-encrypt label -->
<text x="330" y="310" text-anchor="start" font-family="system-ui, sans-serif" font-size="10" fill="rgba(112,240,249,0.7)">re-encrypted</text>
<text x="330" y="322" text-anchor="start" font-family="system-ui, sans-serif" font-size="10" fill="rgba(112,240,249,0.7)">per recipient</text>
<!-- Chunks row (download) -->
<rect x="112" y="336" width="176" height="40" rx="8" fill="none" stroke="#70F0F9" stroke-width="1" stroke-dasharray="4 3"/>
<text x="200" y="361" text-anchor="middle" font-family="system-ui, sans-serif" font-size="12" fill="#70F0F9">encrypted chunks</text>
<!-- Arrow down to recipient -->
<line x1="200" y1="376" x2="200" y2="424" stroke="#70F0F9" stroke-width="1.5" marker-end="url(#arrowC)"/>
<!-- Recipient browser -->
<rect x="120" y="424" width="160" height="56" rx="10" stroke="#70F0F9" stroke-width="1.5"/>
<text x="200" y="448" text-anchor="middle" font-family="system-ui, sans-serif" font-size="13" font-weight="600" fill="#70F0F9">Recipient's browser</text>
<text x="200" y="464" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" fill="rgba(112,240,249,0.7)">decrypts file</text>
<!-- Key path (dashed, side) -->
<path d="M120 44 L8 44 L8 452 L120 452" stroke="#70F0F9" stroke-width="1.5" stroke-dasharray="6 4" fill="none" marker-end="url(#arrowC)"/>
<text x="-6" y="240" text-anchor="middle" font-family="system-ui, sans-serif" font-size="10" fill="#70F0F9" transform="rotate(-90 -6 240)">key in URL fragment - never sent to page server or data router</text>
<!-- Closed padlock: encryption (between sender and chunks) -->
<g transform="translate(192, 88)">
<path d="M4,7 V4 C4,1.2 12,1.2 12,4 V7" stroke="#60a5fa" stroke-width="1.5" fill="none" stroke-linecap="round"/>
<rect x="2" y="7" width="12" height="9" rx="2" fill="#60a5fa"/>
<circle cx="8" cy="12" r="1.2" fill="#0B2A59"/>
</g>
<!-- Open padlock: decryption (between chunks and recipient) -->
<g transform="translate(192, 392)">
<path d="M4,7 V4 C4,1.2 12,1.2 12,4 V2" stroke="#60a5fa" stroke-width="1.5" fill="none" stroke-linecap="round"/>
<rect x="2" y="7" width="12" height="9" rx="2" fill="#60a5fa"/>
<circle cx="8" cy="12" r="1.2" fill="#0B2A59"/>
</g>
<!-- Key icon on dashed line -->
<g transform="translate(8, 410)">
<circle cx="0" cy="0" r="6" stroke="#FBBF24" stroke-width="2" fill="#FBBF24"/>
<circle cx="0" cy="0" r="2" fill="#0B2A59"/>
<line x1="6" y1="0" x2="16" y2="0" stroke="#FBBF24" stroke-width="2"/>
<line x1="14" y1="0" x2="14" y2="4" stroke="#FBBF24" stroke-width="2"/>
<line x1="11" y1="0" x2="11" y2="3.5" stroke="#FBBF24" stroke-width="2"/>
</g>
<!-- Annotation: no shared IDs -->
<text x="200" y="510" text-anchor="middle" font-family="system-ui, sans-serif" font-size="10" fill="rgba(112,240,249,0.7)">Each file fragment uses unique anonymous credentials - no shared identifiers</text>
<defs>
<marker id="arrowC" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#70F0F9"/>
</marker>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 7.2 KiB

@@ -0,0 +1,130 @@
<svg width="440" height="520" viewBox="-20 0 440 520" fill="none" xmlns="http://www.w3.org/2000/svg">
<!-- Sender browser -->
<rect x="120" y="16" width="160" height="56" rx="10" fill="url(#gBox)" stroke="#606C71" stroke-width="1.5"/>
<text x="200" y="40" text-anchor="middle" font-family="system-ui, sans-serif" font-size="13" font-weight="600" fill="#fff">Sender's browser</text>
<text x="200" y="56" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" fill="rgba(255,255,255,0.8)">encrypts file</text>
<!-- Arrow down from sender to chunks -->
<line x1="200" y1="72" x2="200" y2="120" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<!-- Chunks row -->
<rect x="112" y="120" width="176" height="40" rx="8" fill="#f0f7ff" stroke="#0053D0" stroke-width="1" stroke-dasharray="4 3"/>
<text x="200" y="145" text-anchor="middle" font-family="system-ui, sans-serif" font-size="12" fill="#0053D0">encrypted chunks</text>
<!-- Arrows from chunks to routers -->
<line x1="152" y1="160" x2="80" y2="220" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<line x1="200" y1="160" x2="200" y2="220" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<line x1="248" y1="160" x2="320" y2="220" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<!-- Router 1 (SimpleX) -->
<rect x="20" y="220" width="120" height="56" rx="6" fill="#f0f4f8" stroke="#606C71" stroke-width="1.5"/>
<g transform="translate(28, 227)">
<rect width="14" height="4" rx="1" fill="#606C71"/>
<rect y="6" width="14" height="4" rx="1" fill="#606C71"/>
<rect y="12" width="14" height="4" rx="1" fill="#606C71"/>
<circle cx="11" cy="2" r="1" fill="#53C1FF"/>
<circle cx="11" cy="8" r="1" fill="#53C1FF"/>
<circle cx="11" cy="14" r="1" fill="#53C1FF"/>
</g>
<text x="80" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#3F484B">SimpleX</text>
<text x="80" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="#606C71">XFTP router</text>
<!-- Router 2 (Flux) -->
<rect x="155" y="220" width="90" height="56" rx="6" fill="#f0f4f8" stroke="#606C71" stroke-width="1.5"/>
<g transform="translate(163, 227)">
<rect width="14" height="4" rx="1" fill="#606C71"/>
<rect y="6" width="14" height="4" rx="1" fill="#606C71"/>
<rect y="12" width="14" height="4" rx="1" fill="#606C71"/>
<circle cx="11" cy="2" r="1" fill="#53C1FF"/>
<circle cx="11" cy="8" r="1" fill="#53C1FF"/>
<circle cx="11" cy="14" r="1" fill="#53C1FF"/>
</g>
<text x="200" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#3F484B">Flux</text>
<text x="200" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="#606C71">XFTP router</text>
<!-- Router 3 (SimpleX) -->
<rect x="260" y="220" width="120" height="56" rx="6" fill="#f0f4f8" stroke="#606C71" stroke-width="1.5"/>
<g transform="translate(268, 227)">
<rect width="14" height="4" rx="1" fill="#606C71"/>
<rect y="6" width="14" height="4" rx="1" fill="#606C71"/>
<rect y="12" width="14" height="4" rx="1" fill="#606C71"/>
<circle cx="11" cy="2" r="1" fill="#53C1FF"/>
<circle cx="11" cy="8" r="1" fill="#53C1FF"/>
<circle cx="11" cy="14" r="1" fill="#53C1FF"/>
</g>
<text x="320" y="244" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" font-weight="600" fill="#3F484B">SimpleX</text>
<text x="320" y="258" text-anchor="middle" font-family="system-ui, sans-serif" font-size="9" fill="#606C71">XFTP router</text>
<!-- Arrows from routers down -->
<line x1="80" y1="276" x2="152" y2="336" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<line x1="200" y1="276" x2="200" y2="336" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<line x1="320" y1="276" x2="248" y2="336" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<!-- Re-encrypt label -->
<text x="330" y="310" text-anchor="start" font-family="system-ui, sans-serif" font-size="10" fill="#606C71">re-encrypted</text>
<text x="330" y="322" text-anchor="start" font-family="system-ui, sans-serif" font-size="10" fill="#606C71">per recipient</text>
<!-- Chunks row (download) -->
<rect x="112" y="336" width="176" height="40" rx="8" fill="#f0f7ff" stroke="#0053D0" stroke-width="1" stroke-dasharray="4 3"/>
<text x="200" y="361" text-anchor="middle" font-family="system-ui, sans-serif" font-size="12" fill="#0053D0">encrypted chunks</text>
<!-- Arrow down to recipient -->
<line x1="200" y1="376" x2="200" y2="424" stroke="#606C71" stroke-width="1.5" marker-end="url(#arrowG)"/>
<!-- Recipient browser -->
<rect x="120" y="424" width="160" height="56" rx="10" fill="url(#gBox)" stroke="#606C71" stroke-width="1.5"/>
<text x="200" y="448" text-anchor="middle" font-family="system-ui, sans-serif" font-size="13" font-weight="600" fill="#fff">Recipient's browser</text>
<text x="200" y="464" text-anchor="middle" font-family="system-ui, sans-serif" font-size="11" fill="rgba(255,255,255,0.8)">decrypts file</text>
<!-- Key path (dashed, side) -->
<path d="M120 44 L8 44 L8 452 L120 452" stroke="#0053D0" stroke-width="1.5" stroke-dasharray="6 4" fill="none" marker-end="url(#arrowB)"/>
<text x="-6" y="240" text-anchor="middle" font-family="system-ui, sans-serif" font-size="10" fill="#0053D0" transform="rotate(-90 -6 240)">key in URL fragment - never sent to page server or data router</text>
<!-- Closed padlock: encryption (between sender and chunks) -->
<g transform="translate(192, 88)">
<path d="M4,7 V4 C4,1.2 12,1.2 12,4 V7" stroke="#0053D0" stroke-width="1.5" fill="none" stroke-linecap="round"/>
<rect x="2" y="7" width="12" height="9" rx="2" fill="#0053D0"/>
<circle cx="8" cy="12" r="1.2" fill="#fff"/>
</g>
<!-- Open padlock: decryption (between chunks and recipient) -->
<g transform="translate(192, 392)">
<path d="M4,7 V4 C4,1.2 12,1.2 12,4 V2" stroke="#0053D0" stroke-width="1.5" fill="none" stroke-linecap="round"/>
<rect x="2" y="7" width="12" height="9" rx="2" fill="#0053D0"/>
<circle cx="8" cy="12" r="1.2" fill="#fff"/>
</g>
<!-- Key icon on dashed line -->
<g transform="translate(8, 410)">
<circle cx="0" cy="0" r="6" stroke="#D97706" stroke-width="2" fill="#D97706"/>
<circle cx="0" cy="0" r="2" fill="#fff"/>
<line x1="6" y1="0" x2="16" y2="0" stroke="#D97706" stroke-width="2"/>
<line x1="14" y1="0" x2="14" y2="4" stroke="#D97706" stroke-width="2"/>
<line x1="11" y1="0" x2="11" y2="3.5" stroke="#D97706" stroke-width="2"/>
</g>
<!-- Annotation: no shared IDs -->
<text x="200" y="510" text-anchor="middle" font-family="system-ui, sans-serif" font-size="10" fill="#606C71">Each file fragment uses unique anonymous credentials - no shared identifiers</text>
<defs>
<linearGradient id="gBox" x1="120" y1="16" x2="280" y2="72" gradientUnits="userSpaceOnUse">
<stop stop-color="#0053D0"/>
<stop offset="1" stop-color="#53C1FF"/>
</linearGradient>
<linearGradient id="gSrv1" x1="20" y1="220" x2="140" y2="276" gradientUnits="userSpaceOnUse">
<stop stop-color="#0053D0"/>
<stop offset="1" stop-color="#53C1FF"/>
</linearGradient>
<linearGradient id="gSrv2" x1="155" y1="220" x2="245" y2="276" gradientUnits="userSpaceOnUse">
<stop stop-color="#0053D0"/>
<stop offset="1" stop-color="#53C1FF"/>
</linearGradient>
<marker id="arrowG" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#606C71"/>
</marker>
<marker id="arrowB" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#0053D0"/>
</marker>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 7.8 KiB

File diff suppressed because one or more lines are too long
@@ -0,0 +1,145 @@
#app, [data-xftp-app] {
font-family: system-ui, -apple-system, sans-serif;
color: #333;
width: 100%;
max-width: 480px;
padding: 16px;
box-sizing: border-box;
--xftp-ring-fg: #3b82f6;
}
:is(#app, [data-xftp-app]) .card {
background: #fff;
border-radius: 12px;
padding: 32px 24px;
box-shadow: 0 1px 3px rgba(0,0,0,.1);
text-align: center;
}
:is(#app, [data-xftp-app]) h1 {
font-size: 1.25rem;
font-weight: 600;
margin-bottom: 24px;
}
:is(#app, [data-xftp-app]) .stage { margin-top: 16px; }
/* Drop zone */
:is(#app, [data-xftp-app]) .drop-zone {
border: 2px dashed #ccc;
border-radius: 8px;
padding: 32px 16px;
transition: border-color .15s, background .15s;
}
:is(#app, [data-xftp-app]) .drop-zone.drag-over {
border-color: #3b82f6;
background: #eff6ff;
}
/* Buttons */
:is(#app, [data-xftp-app]) .btn {
display: inline-block;
padding: 10px 24px;
border: none;
border-radius: 6px;
background: #3b82f6;
color: #fff;
font-size: .9rem;
font-weight: 500;
cursor: pointer;
transition: background .15s;
}
:is(#app, [data-xftp-app]) .btn:hover { background: #2563eb; }
:is(#app, [data-xftp-app]) .btn-secondary { background: #6b7280; }
:is(#app, [data-xftp-app]) .btn-secondary:hover { background: #4b5563; }
/* Hints */
:is(#app, [data-xftp-app]) .hint { color: #999; font-size: .85rem; margin-top: 8px; }
:is(#app, [data-xftp-app]) .expiry { margin-top: 12px; }
/* Progress */
:is(#app, [data-xftp-app]) .progress-ring { display: block; margin: 0 auto 12px; }
:is(#app, [data-xftp-app]) #upload-status,
:is(#app, [data-xftp-app]) #dl-status { font-size: .9rem; color: #666; margin-bottom: 12px; }
/* Share link row */
:is(#app, [data-xftp-app]) .link-row {
display: flex;
gap: 8px;
margin-top: 12px;
}
:is(#app, [data-xftp-app]) .link-row input {
flex: 1;
padding: 8px 10px;
border: 1px solid #ccc;
border-radius: 6px;
font-size: .85rem;
background: #f9fafb;
}
/* Upload link */
:is(#app, [data-xftp-app]) .upload-link {
margin-top: 12px;
color: #3b82f6;
font-size: .9rem;
text-decoration: none;
cursor: pointer;
}
:is(#app, [data-xftp-app]) .upload-link:not([hidden]) {
display: inline-block;
}
:is(#app, [data-xftp-app]) .upload-link:hover { text-decoration: underline; }
/* Messages */
:is(#app, [data-xftp-app]) .success { color: #16a34a; font-weight: 600; }
:is(#app, [data-xftp-app]) .error { color: #dc2626; font-weight: 500; margin-bottom: 12px; }
/* Security note */
:is(#app, [data-xftp-app]) .security-note {
margin-top: 20px;
padding: 12px;
background: #f0fdf4;
border-radius: 6px;
font-size: .8rem;
color: #555;
text-align: left;
}
:is(#app, [data-xftp-app]) .security-note p + p { margin-top: 6px; }
:is(#app, [data-xftp-app]) .security-note a { color: #3b82f6; text-decoration: none; }
:is(#app, [data-xftp-app]) .security-note a:hover { text-decoration: underline; }
/* ── Dark mode ─────────────────────────────────── */
.dark :is(#app, [data-xftp-app]) {
color: #e5e7eb;
--xftp-ring-bg: #374151;
--xftp-ring-fg: #60a5fa;
--xftp-ring-text: #e5e7eb;
--xftp-ring-done: #4ade80;
}
.dark :is(#app, [data-xftp-app]) .card {
background: #1f2937;
box-shadow: 0 1px 3px rgba(0,0,0,.4);
}
.dark :is(#app, [data-xftp-app]) .drop-zone { border-color: #4b5563; }
.dark :is(#app, [data-xftp-app]) .drop-zone.drag-over {
border-color: #60a5fa;
background: rgba(59,130,246,.15);
}
.dark :is(#app, [data-xftp-app]) .btn-secondary { background: #4b5563; }
.dark :is(#app, [data-xftp-app]) .btn-secondary:hover { background: #374151; }
.dark :is(#app, [data-xftp-app]) .hint { color: #9ca3af; }
.dark :is(#app, [data-xftp-app]) #upload-status,
.dark :is(#app, [data-xftp-app]) #dl-status { color: #9ca3af; }
.dark :is(#app, [data-xftp-app]) .link-row input {
background: #374151;
border-color: #4b5563;
color: #e5e7eb;
}
.dark :is(#app, [data-xftp-app]) .success { color: #4ade80; }
.dark :is(#app, [data-xftp-app]) .error { color: #f87171; }
.dark :is(#app, [data-xftp-app]) .security-note {
background: rgba(34,197,94,.1);
color: #d1d5db;
}
.dark :is(#app, [data-xftp-app]) .upload-link { color: #60a5fa; }
.dark :is(#app, [data-xftp-app]) .security-note a { color: #60a5fa; }
File diff suppressed because one or more lines are too long
+133
View File
@@ -0,0 +1,133 @@
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE GADTs #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE PatternSynonyms #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TupleSections #-}
{-# LANGUAGE TypeApplications #-}
{-# OPTIONS_GHC -fno-warn-ambiguous-fields #-}
module ClientSim
( SimClient (..),
connectClient,
createQueue,
subscribeQueue,
sendMessage,
receiveAndAck,
connectN,
benchKeyHash,
)
where
import Control.Concurrent.Async (mapConcurrently)
import Control.Concurrent.STM
import Control.Monad (forM_)
import Control.Monad.Except (runExceptT)
import Data.ByteString.Char8 (ByteString)
import Data.List (unfoldr)
import qualified Data.List.NonEmpty as L
import Network.Socket (ServiceName)
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Encoding
import Simplex.Messaging.Protocol
import Simplex.Messaging.Transport
import Simplex.Messaging.Transport.Client
import Simplex.Messaging.Version
data SimClient = SimClient
{ scHandle :: THandleSMP TLS 'TClient,
scRcvKey :: C.APrivateAuthKey,
scRcvId :: RecipientId,
scSndId :: SenderId,
scDhSecret :: C.DhSecret 'C.X25519
}
benchKeyHash :: C.KeyHash
benchKeyHash = "LcJUMfVhwD8yxjAiSaDzzGF3-kLG4Uh0Fl_ZIjrRwjI="
connectClient :: TransportHost -> ServiceName -> IO (THandleSMP TLS 'TClient)
connectClient host port = do
let tcConfig = defaultTransportClientConfig {clientALPN = Just alpnSupportedSMPHandshakes}
runTransportClient tcConfig Nothing host port (Just benchKeyHash) $ \h ->
runExceptT (smpClientHandshake h Nothing benchKeyHash supportedClientSMPRelayVRange False Nothing) >>= \case
Right th -> pure th
Left e -> error $ "SMP handshake failed: " <> show e
connectN :: Int -> TransportHost -> ServiceName -> IO [THandleSMP TLS 'TClient]
connectN n host port = do
let batches = chunksOf 100 [1 .. n]
concat <$> mapM (\batch -> mapConcurrently (\_ -> connectClient host port) batch) batches
createQueue :: THandleSMP TLS 'TClient -> IO SimClient
createQueue h = do
g <- C.newRandom
(rPub, rKey) <- atomically $ C.generateAuthKeyPair C.SEd448 g
(sPub, sKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g
(dhPub, dhPriv :: C.PrivateKeyX25519) <- atomically $ C.generateKeyPair g
-- NEW command
Resp "1" NoEntity (Ids rId sId srvDh) <- signSendRecv h rKey ("1", NoEntity, New rPub dhPub)
let dhShared = C.dh' srvDh dhPriv
-- KEY command (secure queue)
Resp "2" _ OK <- signSendRecv h rKey ("2", rId, KEY sPub)
pure SimClient {scHandle = h, scRcvKey = rKey, scRcvId = rId, scSndId = sId, scDhSecret = dhShared}
subscribeQueue :: SimClient -> IO ()
subscribeQueue SimClient {scHandle = h, scRcvKey = rKey, scRcvId = rId} = do
Resp "3" _ (SOK _) <- signSendRecv h rKey ("3", rId, SUB)
pure ()
sendMessage :: THandleSMP TLS 'TClient -> C.APrivateAuthKey -> SenderId -> ByteString -> IO ()
sendMessage h sKey sId body = do
Resp "4" _ OK <- signSendRecv h sKey ("4", sId, SEND noMsgFlags body)
pure ()
receiveAndAck :: SimClient -> IO ()
receiveAndAck SimClient {scHandle = h, scRcvKey = rKey, scRcvId = rId} = do
(_, _, Right (MSG RcvMessage {msgId = mId})) <- tGet1 h
Resp "5" _ OK <- signSendRecv h rKey ("5", rId, ACK mId)
pure ()
-- Helpers (same patterns as ServerTests.hs)
pattern Resp :: CorrId -> EntityId -> BrokerMsg -> Transmission (Either ErrorType BrokerMsg)
pattern Resp corrId queueId command <- (corrId, queueId, Right command)
pattern Ids :: RecipientId -> SenderId -> RcvPublicDhKey -> BrokerMsg
pattern Ids rId sId srvDh <- IDS (QIK rId sId srvDh _ _ Nothing Nothing)
pattern New :: RcvPublicAuthKey -> RcvPublicDhKey -> Command 'Creator
pattern New rPub dhPub = NEW (NewQueueReq rPub dhPub Nothing SMSubscribe (Just (QRMessaging Nothing)) Nothing)
signSendRecv :: (Transport c, PartyI p) => THandleSMP c 'TClient -> C.APrivateAuthKey -> (ByteString, EntityId, Command p) -> IO (Transmission (Either ErrorType BrokerMsg))
signSendRecv h pk t = do
signSend h pk t
(r L.:| _) <- tGetClient h
pure r
signSend :: (Transport c, PartyI p) => THandleSMP c 'TClient -> C.APrivateAuthKey -> (ByteString, EntityId, Command p) -> IO ()
signSend h@THandle {params} (C.APrivateAuthKey a pk) (corrId, qId, cmd) = do
let TransmissionForAuth {tForAuth, tToSend} = encodeTransmissionForAuth params (CorrId corrId, qId, cmd)
authorize t = (,Nothing) <$> case a of
C.SEd25519 -> Just . TASignature . C.ASignature C.SEd25519 $ C.sign' pk t
C.SEd448 -> Just . TASignature . C.ASignature C.SEd448 $ C.sign' pk t
C.SX25519 -> (\THAuthClient {peerServerPubKey = k} -> TAAuthenticator $ C.cbAuthenticate k pk (C.cbNonce corrId) t) <$> thAuth params
Right () <- tPut1 h (authorize tForAuth, tToSend)
pure ()
tPut1 :: Transport c => THandle v c 'TClient -> SentRawTransmission -> IO (Either TransportError ())
tPut1 h t = do
rs <- tPut h (Right t L.:| [])
case rs of
(r : _) -> pure r
[] -> error "tPut1: empty result"
tGet1 :: (ProtocolEncoding v err cmd, Transport c) => THandle v c 'TClient -> IO (Transmission (Either err cmd))
tGet1 h = do
(r L.:| _) <- tGetClient h
pure r
chunksOf :: Int -> [a] -> [[a]]
chunksOf n = unfoldr $ \xs -> if null xs then Nothing else Just (splitAt n xs)
+25
View File
@@ -0,0 +1,25 @@
FROM haskell:9.6.3 AS build
WORKDIR /src
# Copy cabal file first for dependency caching
COPY simplexmq.cabal cabal.project* ./
RUN cabal update && cabal build --only-dependencies -f server_postgres smp-server-bench || true
# Copy full source
COPY . .
RUN cabal build -f server_postgres smp-server-bench \
&& cp $(cabal list-bin -f server_postgres smp-server-bench) /usr/local/bin/smp-server-bench
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
libgmp10 libpq5 libffi8 zlib1g ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /usr/local/bin/smp-server-bench /usr/local/bin/smp-server-bench
COPY tests/fixtures /app/tests/fixtures
WORKDIR /app
ENTRYPOINT ["smp-server-bench"]
+243
View File
@@ -0,0 +1,243 @@
{-# LANGUAGE CPP #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE GADTs #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE NumericUnderscores #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TypeApplications #-}
{-# OPTIONS_GHC -fno-warn-ambiguous-fields #-}
module Main where
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async (async, cancel, forConcurrently_, mapConcurrently, mapConcurrently_)
import Control.Concurrent.STM
import Control.Monad (forever, forM_, void, when)
import Data.ByteString.Char8 (ByteString)
import qualified Data.ByteString.Char8 as B
import Data.IORef
import Data.List (unfoldr)
import Data.Time.Clock (getCurrentTime, utctDayTime)
import Network.Socket (ServiceName)
import System.Environment (getArgs)
import System.IO (hFlush, stdout)
import ClientSim
import Report
import Crypto.Random (ChaChaDRG)
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Server (runSMPServerBlocking)
import Simplex.Messaging.Server.Env.STM as Env
import Simplex.Messaging.Server.Expiration (ExpirationConfig (..))
import Simplex.Messaging.Server.MsgStore.Postgres (PostgresMsgStore)
import Simplex.Messaging.Server.QueueStore.Postgres.Config (PostgresStoreCfg (..))
import Simplex.Messaging.Agent.Store.Postgres.Options (DBOpts (..))
import Simplex.Messaging.Agent.Store.Shared (MigrationConfirmation (..))
import Simplex.Messaging.Client.Agent (SMPClientAgentConfig (..), defaultSMPClientAgentConfig)
import Simplex.Messaging.Transport
import Simplex.Messaging.Transport.Server (ServerCredentials (..), mkTransportServerConfig)
import Simplex.Messaging.Version
import UnliftIO.Exception (bracket)
import Control.Logger.Simple (logInfo, withGlobalLogging, LogConfig (..), setLogLevel, LogLevel (..))
data BenchConfig = BenchConfig
{ numClients :: Int,
sustainedMinutes :: Int,
pgConnStr :: ByteString,
serverPort :: ServiceName,
timeSeriesFile :: FilePath
}
defaultBenchConfig :: BenchConfig
defaultBenchConfig =
BenchConfig
{ numClients = 5000,
sustainedMinutes = 5,
pgConnStr = "postgresql://smp@localhost:15432/smp_bench",
serverPort = "15001",
timeSeriesFile = "bench-timeseries.csv"
}
parseArgs :: IO BenchConfig
parseArgs = do
args <- getArgs
pure $ go args defaultBenchConfig
where
go [] c = c
go ("--clients" : n : rest) c = go rest c {numClients = read n}
go ("--minutes" : n : rest) c = go rest c {sustainedMinutes = read n}
go ("--pg" : s : rest) c = go rest c {pgConnStr = B.pack s}
go ("--port" : p : rest) c = go rest c {serverPort = p}
go ("--timeseries" : f : rest) c = go rest c {timeSeriesFile = f}
go (x : _) _ = error $ "Unknown argument: " <> x
main :: IO ()
main = withGlobalLogging LogConfig {lc_file = Nothing, lc_stderr = True} $ do
setLogLevel LogInfo
bc@BenchConfig {numClients, sustainedMinutes, serverPort, timeSeriesFile, pgConnStr} <- parseArgs
putStrLn $ "SMP Server Memory Benchmark"
putStrLn $ " clients: " <> show numClients
putStrLn $ " sustain: " <> show sustainedMinutes <> " min"
putStrLn $ " pg: " <> B.unpack pgConnStr
putStrLn $ " port: " <> serverPort
putStrLn ""
snapshotsRef <- newIORef []
let snap phase clients = do
s <- takeSnapshot phase clients
modifyIORef' snapshotsRef (s :)
putStrLn $ " [" <> show phase <> "] live=" <> show (snapLive s `div` (1024 * 1024)) <> "MB large=" <> show (snapLarge s `div` (1024 * 1024)) <> "MB"
hFlush stdout
withBenchServer bc $ do
putStrLn "Phase 1: Baseline (no clients)"
snap "baseline" 0
putStrLn $ "Phase 2: Connecting " <> show numClients <> " TLS clients..."
handles <- connectN numClients "localhost" serverPort
putStrLn $ " Connected " <> show (length handles) <> " clients"
snap "tls_connect" (length handles)
putStrLn "Phase 3: Creating queues (NEW + KEY)..."
simClients <- mapConcurrently createQueue handles
putStrLn $ " Created " <> show (length simClients) <> " queues"
snap "queue_create" (length simClients)
putStrLn "Phase 4: Subscribing (SUB)..."
mapConcurrently_ subscribeQueue simClients
snap "subscribe" (length simClients)
-- Pair up clients: first half sends to second half
let halfN = length simClients `div` 2
senders = take halfN simClients
receivers = drop halfN simClients
pairs = zip senders receivers
putStrLn $ "Phase 5: Sending " <> show halfN <> " messages..."
g <- C.newRandom
forConcurrently_ pairs $ \(sender, receiver) -> do
(_, sKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g
sendMessage (scHandle sender) sKey (scSndId receiver) "benchmark test message payload 1234567890"
snap "msg_send" (length simClients)
putStrLn "Phase 6: Receiving and ACKing messages..."
forConcurrently_ receivers receiveAndAck
snap "msg_recv" (length simClients)
putStrLn $ "Phase 7: Sustained load (" <> show sustainedMinutes <> " min)..."
writeTimeSeriesHeader timeSeriesFile
-- Logger thread: snapshot every 10s
logger <- async $ forever $ do
threadDelay 10_000_000
s <- takeSnapshot "sustained" (length simClients)
appendTimeSeries timeSeriesFile s
-- Worker threads: continuous send/receive
let loopDurationUs = sustainedMinutes * 60 * 1_000_000
workersDone <- newTVarIO False
workers <- async $ do
deadline <- (+ loopDurationUs) <$> getMonotonicTimeUs
sustainedLoop g pairs deadline
atomically $ writeTVar workersDone True
-- Wait for workers
void $ atomically $ readTVar workersDone >>= \done -> when (not done) retry
cancel logger
cancel workers
snap "sustained_end" (length simClients)
snapshots <- reverse <$> readIORef snapshotsRef
printSummary snapshots
putStrLn $ "\nTime-series written to: " <> timeSeriesFile
sustainedLoop :: TVar ChaChaDRG -> [(SimClient, SimClient)] -> Int -> IO ()
sustainedLoop g pairs deadline = go
where
go = do
now <- getMonotonicTimeUs
when (now < deadline) $ do
forConcurrently_ pairs $ \(sender, receiver) -> do
(_, sKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g
sendMessage (scHandle sender) sKey (scSndId receiver) "sustained load message payload"
forConcurrently_ (map snd pairs) receiveAndAck
go
getMonotonicTimeUs :: IO Int
getMonotonicTimeUs = do
t <- getCurrentTime
pure $ round (utctDayTime t * 1_000_000)
withBenchServer :: BenchConfig -> IO a -> IO a
withBenchServer BenchConfig {pgConnStr, serverPort} action = do
started <- newEmptyTMVarIO
let srvCfg = benchServerConfig pgConnStr serverPort
bracket
(async $ runSMPServerBlocking started srvCfg Nothing)
cancel
(\_ -> waitForServer started >> action)
where
waitForServer started = do
r <- atomically $ takeTMVar started
if r
then putStrLn $ "Server started on port " <> serverPort
else error "Server failed to start"
benchServerConfig :: ByteString -> ServiceName -> ServerConfig PostgresMsgStore
benchServerConfig pgConn port =
let storeCfg = PostgresStoreCfg
{ dbOpts = DBOpts {connstr = pgConn, schema = "smp_server", poolSize = 10, createSchema = True},
dbStoreLogPath = Nothing,
confirmMigrations = MCYesUp,
deletedTTL = 86400
}
in ServerConfig
{ transports = [(port, transport @TLS, False)],
smpHandshakeTimeout = 120_000_000,
tbqSize = 128,
msgQueueQuota = 128,
maxJournalMsgCount = 256,
maxJournalStateLines = 16,
queueIdBytes = 24,
msgIdBytes = 24,
serverStoreCfg = SSCDatabase storeCfg,
storeNtfsFile = Nothing,
allowNewQueues = True,
newQueueBasicAuth = Nothing,
controlPortUserAuth = Nothing,
controlPortAdminAuth = Nothing,
dailyBlockQueueQuota = 20,
messageExpiration = Just defaultMessageExpiration,
expireMessagesOnStart = False,
expireMessagesOnSend = False,
idleQueueInterval = 14400,
notificationExpiration = defaultNtfExpiration,
inactiveClientExpiration = Nothing,
logStatsInterval = Nothing,
logStatsStartTime = 0,
serverStatsLogFile = "bench/tmp/stats.log",
serverStatsBackupFile = Nothing,
prometheusInterval = Nothing,
prometheusMetricsFile = "bench/tmp/metrics.txt",
pendingENDInterval = 500_000,
ntfDeliveryInterval = 200_000,
smpCredentials =
ServerCredentials
{ caCertificateFile = Just "tests/fixtures/ca.crt",
privateKeyFile = "tests/fixtures/server.key",
certificateFile = "tests/fixtures/server.crt"
},
httpCredentials = Nothing,
smpServerVRange = supportedServerSMPRelayVRange,
Env.transportConfig = mkTransportServerConfig True (Just alpnSupportedSMPHandshakes) True,
controlPort = Nothing,
smpAgentCfg = defaultSMPClientAgentConfig {persistErrorInterval = 1},
allowSMPProxy = False,
serverClientConcurrency = 16,
information = Nothing,
startOptions = StartOptions {maintenance = False, compactLog = False, logLevel = LogInfo, skipWarnings = True, confirmMigrations = MCYesUp}
}
+113
View File
@@ -0,0 +1,113 @@
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE NumericUnderscores #-}
{-# LANGUAGE OverloadedStrings #-}
module Report
( Snapshot (..),
takeSnapshot,
printSummary,
writeTimeSeriesHeader,
appendTimeSeries,
)
where
import Control.Concurrent (threadDelay)
import Data.List (foldl')
import Data.Text (Text)
import qualified Data.Text as T
import qualified Data.Text.IO as T
import Data.Time.Clock (UTCTime, getCurrentTime)
import Data.Time.Format.ISO8601 (iso8601Show)
import Data.Word (Word32, Word64)
import GHC.Stats (RTSStats (..), GCDetails (..), getRTSStats)
import System.IO (Handle, IOMode (..), hFlush, hSetBuffering, BufferMode (..), withFile)
import System.Mem (performMajorGC)
data Snapshot = Snapshot
{ snapTime :: UTCTime,
snapPhase :: Text,
snapLive :: Word64,
snapHeap :: Word64,
snapLarge :: Word64,
snapFrag :: Word64,
snapGCs :: Word32,
snapClients :: Int
}
takeSnapshot :: Text -> Int -> IO Snapshot
takeSnapshot phase clients = do
performMajorGC
threadDelay 1_000_000
rts <- getRTSStats
ts <- getCurrentTime
let GCDetails {gcdetails_live_bytes, gcdetails_mem_in_use_bytes, gcdetails_large_objects_bytes, gcdetails_block_fragmentation_bytes} = gc rts
pure
Snapshot
{ snapTime = ts,
snapPhase = phase,
snapLive = gcdetails_live_bytes,
snapHeap = gcdetails_mem_in_use_bytes,
snapLarge = gcdetails_large_objects_bytes,
snapFrag = gcdetails_block_fragmentation_bytes,
snapGCs = gcs rts,
snapClients = clients
}
printSummary :: [Snapshot] -> IO ()
printSummary [] = putStrLn "No snapshots collected."
printSummary snaps = do
putStrLn ""
putStrLn hdr
putStrLn $ replicate (length hdr) '-'
mapM_ printRow (zip (Snapshot {snapLive = 0, snapHeap = 0, snapLarge = 0, snapFrag = 0, snapGCs = 0, snapClients = 0, snapPhase = "", snapTime = snapTime (head snaps)} : snaps) snaps)
where
hdr = padR 20 "Phase" <> padL 12 "live_MB" <> padL 12 "large_MB" <> padL 12 "frag_MB" <> padL 12 "heap_MB" <> padL 10 "clients" <> padL 14 "d_live_MB" <> padL 14 "d_large_MB" <> padL 14 "KB/client"
printRow (prev, cur) =
putStrLn $
padR 20 (T.unpack $ snapPhase cur)
<> padL 12 (showMB $ snapLive cur)
<> padL 12 (showMB $ snapLarge cur)
<> padL 12 (showMB $ snapFrag cur)
<> padL 12 (showMB $ snapHeap cur)
<> padL 10 (show $ snapClients cur)
<> padL 14 (showDeltaMB (snapLive cur) (snapLive prev))
<> padL 14 (showDeltaMB (snapLarge cur) (snapLarge prev))
<> padL 14 (perClient cur)
showMB w = show (w `div` (1024 * 1024))
showDeltaMB a b
| a >= b = "+" <> show ((a - b) `div` (1024 * 1024))
| otherwise = "-" <> show ((b - a) `div` (1024 * 1024))
perClient Snapshot {snapClients, snapLive}
| snapClients > 0 = show (snapLive `div` fromIntegral snapClients `div` 1024)
| otherwise = "-"
padR n s = s <> replicate (max 0 (n - length s)) ' '
padL n s = replicate (max 0 (n - length s)) ' ' <> s
csvHeader :: Text
csvHeader = "timestamp,phase,rts_live,rts_heap,rts_large,rts_frag,rts_gc,clients"
snapshotCsv :: Snapshot -> Text
snapshotCsv Snapshot {snapTime, snapPhase, snapLive, snapHeap, snapLarge, snapFrag, snapGCs, snapClients} =
T.intercalate
","
[ T.pack $ iso8601Show snapTime,
snapPhase,
tshow snapLive,
tshow snapHeap,
tshow snapLarge,
tshow snapFrag,
tshow snapGCs,
tshow snapClients
]
writeTimeSeriesHeader :: FilePath -> IO ()
writeTimeSeriesHeader path = T.writeFile path (csvHeader <> "\n")
appendTimeSeries :: FilePath -> Snapshot -> IO ()
appendTimeSeries path snap =
withFile path AppendMode $ \h -> do
hSetBuffering h LineBuffering
T.hPutStrLn h $ snapshotCsv snap
tshow :: Show a => a -> Text
tshow = T.pack . show
+46
View File
@@ -0,0 +1,46 @@
services:
postgres:
image: postgres:17
environment:
POSTGRES_USER: smp
POSTGRES_DB: smp_bench
POSTGRES_HOST_AUTH_METHOD: trust
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U smp -d smp_bench"]
interval: 2s
timeout: 5s
retries: 10
bench:
build:
context: ..
dockerfile: bench/Dockerfile
depends_on:
postgres:
condition: service_healthy
environment:
BENCH_PG: "postgresql://smp@postgres/smp_bench"
BENCH_CLIENTS: "${BENCH_CLIENTS:-5000}"
BENCH_MINUTES: "${BENCH_MINUTES:-5}"
command:
- "--pg"
- "postgresql://smp@postgres/smp_bench"
- "--clients"
- "${BENCH_CLIENTS:-5000}"
- "--minutes"
- "${BENCH_MINUTES:-5}"
- "--timeseries"
- "/results/timeseries.csv"
- "+RTS"
- "-N"
- "-A16m"
- "-T"
- "-RTS"
volumes:
- ./results:/results
volumes:
pgdata:
+2
View File
@@ -0,0 +1,2 @@
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE SCHEMA IF NOT EXISTS smp_server;
Executable
+60
View File
@@ -0,0 +1,60 @@
#!/bin/bash
set -e
cd "$(dirname "$0")"
mkdir -p results
reset_db() {
docker compose down -v 2>/dev/null || true
docker compose up -d --wait postgres
echo "PostgreSQL ready."
}
if [ "$1" = "--compare-rts" ]; then
shift
docker compose build bench
for label_flags in \
"default:-N -A16m -T" \
"F1.2:-N -A16m -F1.2 -T" \
"F1.5:-N -A16m -F1.5 -T" \
"A4m:-N -A4m -T" \
"A4m-F1.2:-N -A4m -F1.2 -T" \
"compact:-N -A16m -c -T" \
"nonmoving:-N -A16m -xn -T"; do
label="${label_flags%%:*}"
flags="${label_flags#*:}"
echo ""
echo "=========================================="
echo " RTS config: $label ($flags)"
echo "=========================================="
reset_db
docker compose run --rm \
-e BENCH_CLIENTS="${BENCH_CLIENTS:-1000}" \
-e BENCH_MINUTES="${BENCH_MINUTES:-2}" \
bench \
--pg "postgresql://smp@postgres/smp_bench" \
--clients "${BENCH_CLIENTS:-1000}" \
--minutes "${BENCH_MINUTES:-2}" \
--timeseries "/results/bench-${label}.csv" \
"$@" \
+RTS $flags -RTS
done
echo ""
echo "Done. Results in bench/results/"
elif [ "$1" = "--local" ]; then
# Run natively (not in container) — requires local Postgres
shift
reset_db
cabal run smp-server-bench -f server_postgres -- \
--pg "postgresql://smp@localhost:15432/smp_bench" \
--clients "${BENCH_CLIENTS:-5000}" \
--minutes "${BENCH_MINUTES:-5}" \
"$@" \
+RTS -N -A16m -s -RTS
else
# Run fully in containers
reset_db
docker compose run --rm bench "$@"
fi
docker compose down
+23 -6
View File
@@ -4,6 +4,15 @@ packages: .
-- packages: . ../http2
-- packages: . ../network-transport
-- uncomment two sections below to run tests with coverage
-- package *
-- coverage: True
-- library-coverage: True
-- package attoparsec
-- coverage: False
-- library-coverage: False
index-state: 2023-12-12T00:00:00Z
package cryptostore
@@ -14,12 +23,6 @@ source-repository-package
location: https://github.com/simplex-chat/aeson.git
tag: aab7b5a14d6c5ea64c64dcaee418de1bb00dcc2b
-- old bs/text compat for 8.10
source-repository-package
type: git
location: https://github.com/simplex-chat/base64.git
tag: 2d77b6dbcaffc00570a70be8694049f3710e7c94
source-repository-package
type: git
location: https://github.com/simplex-chat/hs-socks.git
@@ -34,3 +37,17 @@ source-repository-package
type: git
location: https://github.com/simplex-chat/sqlcipher-simple.git
tag: a46bd361a19376c5211f1058908fc0ae6bf42446
-- waiting for published warp-tls-3.4.7
source-repository-package
type: git
location: https://github.com/yesodweb/wai.git
tag: ec5e017d896a78e787a5acea62b37a4e677dec2e
subdir: warp-tls
-- backported fork due http-5.0
source-repository-package
type: git
location: https://github.com/simplex-chat/wai.git
tag: 2f6e5aa5f05ba9140ac99e195ee647b4f7d926b0
subdir: warp
+104
View File
@@ -0,0 +1,104 @@
# Coding and building
This file provides guidance on coding style and approaches and on building the code.
## Code Security
When designing code and planning implementations:
- Apply adversarial thinking, and consider what may happen if one of the communicating parties is malicious.
- Formulate an explicit threat model for each change - who can do which undesirable things and under which circumstances.
## Code Quality Standards
Haskell client and server code serves as system specification, not just implementation — we use type-driven design to reflect the business domain in types. Quality, conciseness, and clarity of Haskell code are critical.
## Code Style, Formatting and Approaches
The project uses **fourmolu** for Haskell code formatting. Configuration is in `fourmolu.yaml`.
**Key formatting rules:**
- 2-space indentation
- Trailing function arrows, commas, and import/export style
- Record brace without space: `{field = value}`
- Single newline between declarations
- Never use unicode symbols
- Inline `let` style with right-aligned `in`
**Format code before committing:**
```bash
# Format a single file
fourmolu -i src/Simplex/Messaging/Protocol.hs
```
Some files that use CPP language extension cannot be formatted as a whole, so individual code fragments need to be formatted.
**Follow existing code patterns:**
- Match the style of surrounding code
- Use qualified imports with short aliases (e.g., `import qualified Data.ByteString.Char8 as B`)
- Use record syntax for types with multiple fields
- Prefer explicit pattern matching over partial functions
**Comments policy:**
- Avoid redundant comments that restate what the code already says
- Only comment on non-obvious design decisions or tricky implementation details
- Function names and type signatures should be self-documenting
- Do not add comments like "wire format encoding" (Encoding class is always wire format) or "check if X" when the function name already says that
- Assume a competent Haskell reader
**Diff and refactoring:**
- Avoid unnecessary changes and code movements
- Never do refactoring unless it substantially reduces cost of solving the current problem, including the cost of refactoring
- Aim to minimize the code changes - do what is minimally required to solve users' problems
**Document and code structure:**
- **Never move existing code or sections around** - add new content at appropriate locations without reorganizing existing structure.
- When adding new sections to documents, continue the existing numbering scheme.
- Minimize diff size - prefer small, targeted changes over reorganization.
**Code analysis and review:**
- Trace data flows end-to-end: from origin, through storage/parameters, to consumption. Flag values that are discarded and reconstructed from partial data (e.g. extracted from a URI missing original fields) — this is usually a bug.
- Read implementations of called functions, not just signatures — if duplication involves a called function, check whether decomposing it resolves the duplication.
- Do not save time on analysis. Read every function in the data flow even when the interface seems clear — wrong assumptions about internals are the main source of missed bugs.
### Haskell Extensions
- `StrictData` enabled by default
- Use STM for safe concurrency
- Assume concurrency in PostgreSQL queries
- Comprehensive warning flags with strict pattern matching
## Build Commands
```bash
# Standard build
cabal build
# Fast build
cabal build --ghc-options -O0
# Build specific executables
cabal build exe:smp-server exe:xftp-server exe:ntf-server exe:xftp
# Build with PostgreSQL server support
cabal build -fserver_postgres
# Client-only library build (no server code)
cabal build -fclient_library
# Find binary location
cabal list-bin exe:smp-server
```
### Cabal Flags
- `swift`: Enable Swift JSON format
- `client_library`: Build without server code
- `client_postgres`: Use PostgreSQL instead of SQLite for agent persistence
- `server_postgres`: PostgreSQL support for server queue/notification store
## External Dependencies
Custom forks specified in `cabal.project`:
- `aeson`, `hs-socks` (SimpleX forks)
- `direct-sqlcipher`, `sqlcipher-simple` (encrypted SQLite)
- `warp`, `warp-tls` (HTTP server)
+105
View File
@@ -0,0 +1,105 @@
# SimpleXMQ repository
This file provides guidance on the project structure to help working with code in this repository.
## Project Overview
SimpleXMQ is a Haskell message broker implementing unidirectional (simplex) queues for privacy-preserving messaging.
Key components:
- **SimpleX Messaging Protocol**: SMP protocol definition and encodings ([code](../src/Simplex/Messaging/Protocol.hs), [transport code](../src/Simplex/Messaging/Transport.hs), [spec](../protocol/simplex-messaging.md)).
- **SMP Server**: Message broker with TLS, in-memory queues, optional persistence ([main code](../src/Simplex/Messaging/Server.hs), [all code files](../src/Simplex/Messaging/Server/), [executable](../apps/smp-server/)). For proxying SMP commands the server uses [lightweight SMP client](../src/Simplex/Messaging/Client/Agent.hs).
- **SMP Client**: Functional API with STM-based message delivery ([code](../src/Simplex/Messaging/Client.hs)).
- **SMP Agent**: High-level duplex connections via multiple simplex queues with E2E encryption ([code](../src/Simplex/Messaging/Agent.hs)). Implements Agent-to-agent protocol ([code](../src/Simplex/Messaging/Agent/Protocol.hs), [spec](../protocol/agent-protocol.md)) via intermediary agent client ([code](../src/Simplex/Messaging/Agent/Client.hs)).
- **XFTP**: SimpleX File Transfer Protocol, server and CLI client ([code](../src/Simplex/FileTransfer/), [spec](../protocol/xftp.md)).
- **XRCP**: SimpleX Remote Control Protocol ([code](`../src/Simplex/RemoteControl/`), [spec](../protocol/xrcp.md)).
- **Notifications**: Push notifications server requires PostgreSQL ([code](../src/Simplex/Messaging/Notifications), [executable](../apps/ntf-server/)). Client protocol is used for clients to communicate with the server ([code](../src/Simplex/Messaging/Notifications/Protocol.hs), [spec](../protocol/push-notifications.md)). For subscribing to SMP notifications the server uses [lightweight SMP client](../src/Simplex/Messaging/Client/Agent.hs).
## Architecture
For general overview see `../protocol/overview-tjr.md`.
SMP Protocol Layers:
```
TLS Transport → SMP Protocol → Agent Protocol → Application protocol
```
XFTP Protocol Layers:
```
TLS Transport (HTTP2 encoding) → XFTP Protocol → Out-of-band file descriptions
```
## Key Patterns
1. **Persistence**: All queue state managed via Software Transactional Memory or via PostgreSQL
- `Simplex.Messaging.Server.MsgStore.STM` - in-memory messages
- `Simplex.Messaging.Server.QueueStore.STM` - in-memory queue state
- `Simplex.Messaging.Server.MsgStore.Postgres` - message storage
- `Simplex.Messaging.Server.QueueStore.Postgres` - queue storage
2. **Append-Only Store Log**: Optional persistence via journal for in-memory storage
- `Simplex.Messaging.Server.StoreLog` - queue creation log
- Compacted on restart
3. **Agent Storage**:
- SQLite (default) or PostgreSQL
- Migrations in `src/Simplex/Messaging/Agent/Store/{SQLite,Postgres}/Migrations/`
4. **Protocol Versioning**: All layers support version negotiation
- `Simplex.Messaging.Version` - version range utilities
5. **Double Ratchet E2E**: Per-connection encryption
- `Simplex.Messaging.Crypto.Ratchet`
- SNTRUP761 post-quantum KEM (`src/Simplex/Messaging/Crypto/SNTRUP761/`)
## Source Layout
```
src/Simplex/
├── Messaging/
│ ├── Agent.hs # Main agent (~210KB)
│ ├── Server.hs # SMP server (~130KB)
│ ├── Client.hs # Client API (~65KB)
│ ├── Protocol.hs # Protocol types (~77KB)
│ ├── Crypto.hs # E2E encryption (~52KB)
│ ├── Transport.hs # Transport encoding over TLS
│ ├── Agent/Store/ # SQLite/Postgres persistence
│ ├── Server/ # Server internals (QueueStore, MsgStore, Control)
│ └── Notifications/ # Push notification system
├── FileTransfer/ # XFTP implementation for file transfers
└── RemoteControl/ # XRCP implementation for device discovery & control
```
## Protocol Documentation
- `protocol/overview-tjr.md`: SMP protocols stack overview
- `protocol/simplex-messaging.md`: SMP protocol spec (v19)
- `protocol/agent-protocol.md`: Agent protocol spec (v7)
- `protocol/xftp.md`: File transfer protocol
- `protocol/xrcp.md`: Remote control protocol
- `rfcs/`: Design RFCs for features
## Testing
```bash
# Run all tests
cabal test --test-show-details=streaming
# Run specific test group (uses HSpec)
cabal test --test-option=--match="/Core tests/Encryption tests/"
# Run single test
cabal test --test-option=--match="/SMP client agent/functional API/"
```
Tests require PostgreSQL running on `localhost:5432` when using `-fserver_postgres` or `-fclient_postgres`.
Test files are in `tests/` with structure:
- `Test.hs`: Main runner
- `AgentTests/`: Agent protocol and connection tests
- `CoreTests/`: Crypto, encoding, storage tests
- `ServerTests.hs`: SMP server tests
- `XFTPServerTests.hs`: File transfer tests
+23
View File
@@ -0,0 +1,23 @@
# Contributing to SimpleX repositories
## Focus on user problems
We do not make code changes to improve code - any change must address a specific user problem or request.
## Discuss the plans as early as possible
Please discuss the problem you want to solve and your detailed implementation plan with the project team prior to contributing, to avoid wasted time and additional changes. Acceptance of your contribution depends on your willingness and ability to iterate the proposed contribution to achieve the required quality level, coding style, test coverage, and alignment with user requirements as they are understood by the project team.
## Follow project structure, coding style and approaches
./PROJECT.md has information about the structure of this `simplexmq` repository.
./CODE.md has details about general requirements common for `simplexmq` and `simplex-chat` repositories.
This files can be used with LLM prompts, e.g. if you use Claude Code you can create CLAUDE.md file in project root importing content from these files:
```markdown
@README.md
@contributing/PROJECT.md
@contributing/CODE.md
```
+100
View File
@@ -0,0 +1,100 @@
## Memory Diagnostics Results
### Data Collection
Server: smp19.simplex.im, PostgreSQL backend, `useCache = False`
RTS flags: `+RTS -N -A16m -I0.01 -Iw15 -s -RTS` (16 cores)
### Mar 20 Data (1 hour, 07:19-08:19)
```
Time rts_live rts_heap rts_large rts_frag clients non-large
07:19 7.5 GB 8.2 GB 5.5 GB 0.03 GB 14,000 2.0 GB
07:24 6.4 GB 10.8 GB 5.2 GB 3.6 GB 14,806 1.2 GB
07:29 8.2 GB 10.8 GB 6.5 GB 1.8 GB 15,667 1.7 GB
07:34 10.0 GB 12.3 GB 7.9 GB 1.4 GB 15,845 2.1 GB
07:39 6.7 GB 13.0 GB 5.3 GB 5.6 GB 16,589 1.4 GB
07:44 8.5 GB 13.0 GB 6.7 GB 3.7 GB 16,283 1.8 GB
07:49 6.5 GB 13.0 GB 5.2 GB 5.8 GB 16,532 1.3 GB
07:54 6.0 GB 13.0 GB 4.8 GB 6.3 GB 16,636 1.2 GB
07:59 6.4 GB 13.0 GB 5.1 GB 5.9 GB 16,769 1.3 GB
08:04 8.3 GB 13.0 GB 6.5 GB 3.9 GB 17,352 1.8 GB
08:09 10.2 GB 13.0 GB 8.0 GB 1.9 GB 17,053 2.2 GB
08:14 5.6 GB 13.0 GB 4.5 GB 6.8 GB 17,147 1.1 GB
08:19 7.6 GB 13.0 GB 6.1 GB 4.6 GB 17,496 1.5 GB
```
non-large = rts_live - rts_large (normal Haskell heap objects: Maps, TVars, closures)
### Mar 19 Data (5.5 hours, 07:49-13:19)
rts_heap grew from 10.1 GB to 20.7 GB over 5.5 hours.
Post-GC rts_live floor rose from 5.5 GB to 9.1 GB.
### Findings
**1. Large/pinned objects dominate live data (60-80%)**
`rts_large` = 4.5-8.0 GB out of 5.6-10.2 GB live. These are allocations > ~3KB that go on GHC's large object heap. They oscillate (not growing monotonically), meaning they are being allocated and freed constantly — transient, not leaked.
**2. Fragmentation is the heap growth mechanism**
`rts_heap ≈ rts_live + rts_frag`. The heap grows because pinned/large objects fragment GHC's block allocator. Once GHC expands the heap, it never shrinks. Growth pattern:
- Large objects allocated → occupy blocks
- Large objects freed → blocks can't be reused if ANY other object shares the block
- New allocations need fresh blocks → heap expands
- Heap never returns memory to OS
**3. Non-large heap data is stable (~1.0-2.2 GB)**
Normal Haskell objects (Maps, TVars, closures, client structures) account for only 1-2 GB. This scales with client count at ~100-130 KB/client and does NOT grow over time.
**4. All tracked data structures are NOT the cause**
- `clientSndQ=0, clientMsgQ=0` — TBQueues empty, no message accumulation
- `smpQSubs` oscillates ~1.0-1.4M — entries are cleaned up, not leaking
- `ntfStore` < 2K entries — negligible
- All proxy agent maps near 0
- `loadedQ=0` — useCache=False confirmed working
**5. Source of large objects is unclear without heap profiling**
The 4.5-8.0 GB of large objects could come from:
- PostgreSQL driver (`postgresql-simple`/`libpq`) — pinned ByteStrings for query results
- TLS library (`tls`) — pinned buffers per connection
- Network socket I/O — pinned ByteStrings for recv/send
- SMP protocol message blocks
Cannot distinguish between these without `-hT` heap profiling (which is too expensive for this server).
### Root Cause
**GHC heap fragmentation from constant churn of large/pinned ByteString allocations.**
Not a data structure leak. The live data itself is reasonable (5-10 GB for 15-17K clients). The problem is that GHC's copying GC cannot compact around pinned objects, so the heap grows with fragmentation and never shrinks.
### Mitigation Options
All are RTS flag changes — no rebuild needed, reversible by restart.
**1. `-F1.2`** (reduce GC trigger factor from default 2.0)
- Triggers major GC when heap reaches 1.2x live data instead of 2x
- Reclaims fragmented blocks sooner
- Trade-off: more frequent GC, slightly higher CPU
- Risk: low — just makes GC run more often
**2. Reduce `-A16m` to `-A4m`** (smaller nursery)
- More frequent minor GC → short-lived pinned objects freed faster
- Trade-off: more GC cycles, but each is smaller
- Risk: low — may actually improve latency by reducing GC pause times
**3. `+RTS -xn`** (nonmoving GC)
- Designed for pinned-heavy workloads — avoids copying entirely
- Available since GHC 8.10, improved in 9.x
- Trade-off: different GC characteristics, less battle-tested
- Risk: medium — different GC algorithm, should test first
**4. Limit concurrent connections** (application-level)
- Since large objects scale per-client, fewer clients = less fragmentation
- Trade-off: reduced capacity
- Risk: low but impacts users
+225
View File
@@ -0,0 +1,225 @@
## Root Cause Analysis: SMP Server Memory Growth (23.5GB)
### Environment
- **Server**: smp19.simplex.im, ~21,927 connected clients
- **Storage**: PostgreSQL backend with `useCache = False`
- **RTS flags**: `+RTS -N -A16m -I0.01 -Iw15 -s -RTS` (16 cores)
- **Memory**: 23.5GB RES / 1031GB VIRT (75% of available RAM)
### Log Summary
- **Duration**: ~22 hours (Mar 16 12:12 → Mar 17 10:20)
- **92,277 proxy connection errors** out of 92,656 total log lines (99.6%)
- **292 unique failing destination servers**, top offender: `nowhere.moe` (12,875 errors)
- Only **145 successful proxy connections**
---
### Known Factor: GHC Heap Sizing
With 16 cores and `-A16m`:
- **Nursery**: 16 × 16MB = **256MB baseline**
- GHC default major GC threshold = **2× live data** — if live data is 10GB, heap grows to ~20GB before major GC
- The server is rarely idle with 22K clients, so major GC is deferred despite `-I0.01`
- This is an amplifier — whatever the actual live data size is, GHC roughly doubles it
---
### Candidate Structures That Could Grow Unboundedly
Analysis of the full codebase identified these structures that either grow without bound or have uncertain cleanup:
#### 1. `SubscribedClients` maps — `Env/STM.hs:378`
Both `subscribers.queueSubscribers` and `ntfSubscribers.queueSubscribers` (and their `serviceSubscribers`) use `SubscribedClients (TMap EntityId (TVar (Maybe (Client s))))`.
Comment at line 376: *"The subscriptions that were made at any point are not removed"*
`deleteSubcribedClient` IS called on disconnect (Server.hs:1112) and DOES call `TM.delete`. But it only deletes if the current stored client matches — if another client already re-subscribed, the old client's disconnect won't remove the entry. This is by design for mobile client continuity, but the net effect on map size over time is unclear without measurement.
#### 2. ProxyAgent's subscription TMaps — `Client/Agent.hs:145-151`
The `SMPClientAgent` has 4 TMaps that accumulate one top-level entry per unique destination server and **never remove** them:
- `activeServiceSubs :: TMap SMPServer (TVar ...)` (line 145)
- `activeQueueSubs :: TMap SMPServer (TMap QueueId ...)` (line 146)
- `pendingServiceSubs :: TMap SMPServer (TVar ...)` (line 149)
- `pendingQueueSubs :: TMap SMPServer (TMap QueueId ...)` (line 150)
Comment at line 262: *"these vars are never removed, they are only added"*
These are only used for the proxy agent (SParty 'Sender), so they grow with each unique destination SMP server proxied to. With 292 unique servers in this log period, these are likely small — but long-running servers may accumulate thousands.
`closeSMPClientAgent` (line 369) does NOT clear these 4 maps.
#### 3. `NtfStore``NtfStore.hs:26`
`NtfStore (TMap NotifierId (TVar [MsgNtf]))` — one entry per NotifierId.
`deleteExpiredNtfs` (line 47) filters expired notifications from lists but does **not remove entries with empty lists** from the TMap. Over time, NotifierIds that no longer receive notifications leave zombie `TVar []` entries.
`deleteNtfs` (line 44) does remove the full entry via `TM.lookupDelete` — but only called when a notifier is explicitly deleted.
#### 4. `serviceLocks` in PostgresQueueStore — `Postgres.hs:112,469`
`serviceLocks :: TMap CertFingerprint Lock` — one Lock per unique certificate fingerprint.
`getCreateService` (line 469) calls `withLockMap (serviceLocks st) fp` which calls `getMapLock` (Agent/Client.hs:1029-1032) — this **unconditionally inserts** a Lock into the TMap. There is **no cleanup code** for serviceLocks anywhere. This is NOT guarded by `useCache`.
#### 5. `sentCommands` per proxy client connection — `Client.hs:580`
Each `PClient` has `sentCommands :: TMap CorrId (Request err msg)`. Entries are added per command sent (line 1369) and only removed when a response arrives (line 698). If a connection drops before all responses arrive, entries remain until the `PClient` is GC'd. Since `PClient` is captured by the connection thread which terminates on error, the `PClient` should become GC-eligible — but GC timing depends on heap pressure.
#### 6. `subQ :: TQueue (ClientSub, ClientId)``Env/STM.hs:363`
Unbounded `TQueue` for subscription changes. If the subscriber thread (`serverThread`) can't process changes fast enough, this queue grows without backpressure. With 22K clients subscribing/unsubscribing, sustained bursts could cause this queue to bloat.
---
### Ruled Out
1. **PostgreSQL queue cache**: `useCache = False``queues`, `senders`, `links`, `notifiers` TMaps are empty.
2. **`notifierLocks`**: Guarded by `useCache` (Postgres.hs:377,405) — not used with `useCache = False`.
3. **Client structures**: 22K × ~3KB = ~66MB — negligible.
4. **TBQueues**: Bounded (`tbqSize = 128`).
5. **Thread management**: `forkClient` uses weak refs + `finally` blocks. `endThreads` cleared on disconnect.
6. **Proxy `smpClients`/`smpSessions`**: Properly cleaned on disconnect/expiry.
7. **`smpSubWorkers`**: Properly cleaned on worker completion; also cleared in `closeSMPClientAgent`.
8. **`pendingEvents`**: Atomically swapped empty every `pendingENDInterval`.
9. **Stats IORef counters**: Fixed number, bounded.
10. **DB connection pool**: Bounded `TBQueue` with bracket-based return.
---
### Insufficient Data to Determine Root Cause
Without measuring the actual sizes of these structures at runtime, we cannot determine which (if any) is the primary contributor. The following exact logging changes will identify the root cause.
---
### EXACT LOGS TO ADD
Add a new periodic logging thread in `src/Simplex/Messaging/Server.hs`.
Insert at `Server.hs:197` (after `prometheusMetricsThread_`):
```haskell
<> memoryDiagThread_ cfg
```
Then define:
```haskell
memoryDiagThread_ :: ServerConfig s -> [M s ()]
memoryDiagThread_ ServerConfig {prometheusInterval = Just _} =
[memoryDiagThread]
memoryDiagThread_ _ = []
memoryDiagThread :: M s ()
memoryDiagThread = do
labelMyThread "memoryDiag"
Env { ntfStore = NtfStore ntfMap
, server = srv@Server {subscribers, ntfSubscribers}
, proxyAgent = ProxyAgent {smpAgent = pa}
, msgStore_ = ms
} <- ask
let interval = 300_000_000 -- 5 minutes
liftIO $ forever $ do
threadDelay interval
-- GHC RTS stats
rts <- getRTSStats
let liveBytes = gcdetails_live_bytes $ gc rts
heapSize = gcdetails_mem_in_use_bytes $ gc rts
gcCount = gcs rts
-- Server structures
clientCount <- IM.size <$> getServerClients srv
-- SubscribedClients (queue and service subscribers for both SMP and NTF)
smpQSubs <- M.size <$> getSubscribedClients (queueSubscribers subscribers)
smpSSubs <- M.size <$> getSubscribedClients (serviceSubscribers subscribers)
ntfQSubs <- M.size <$> getSubscribedClients (queueSubscribers ntfSubscribers)
ntfSSubs <- M.size <$> getSubscribedClients (serviceSubscribers ntfSubscribers)
-- Pending events
smpPending <- IM.size <$> readTVarIO (pendingEvents subscribers)
ntfPending <- IM.size <$> readTVarIO (pendingEvents ntfSubscribers)
-- NtfStore
ntfStoreSize <- M.size <$> readTVarIO ntfMap
-- ProxyAgent maps
let SMPClientAgent {smpClients, smpSessions, activeServiceSubs, activeQueueSubs, pendingServiceSubs, pendingQueueSubs, smpSubWorkers} = pa
paClients <- M.size <$> readTVarIO smpClients
paSessions <- M.size <$> readTVarIO smpSessions
paActSvc <- M.size <$> readTVarIO activeServiceSubs
paActQ <- M.size <$> readTVarIO activeQueueSubs
paPndSvc <- M.size <$> readTVarIO pendingServiceSubs
paPndQ <- M.size <$> readTVarIO pendingQueueSubs
paWorkers <- M.size <$> readTVarIO smpSubWorkers
-- Loaded queue counts
lc <- loadedQueueCounts $ fromMsgStore ms
-- Log everything
logInfo $
"MEMORY "
<> "rts_live=" <> tshow liveBytes
<> " rts_heap=" <> tshow heapSize
<> " rts_gc=" <> tshow gcCount
<> " clients=" <> tshow clientCount
<> " smpQSubs=" <> tshow smpQSubs
<> " smpSSubs=" <> tshow smpSSubs
<> " ntfQSubs=" <> tshow ntfQSubs
<> " ntfSSubs=" <> tshow ntfSSubs
<> " smpPending=" <> tshow smpPending
<> " ntfPending=" <> tshow ntfPending
<> " ntfStore=" <> tshow ntfStoreSize
<> " paClients=" <> tshow paClients
<> " paSessions=" <> tshow paSessions
<> " paActSvc=" <> tshow paActSvc
<> " paActQ=" <> tshow paActQ
<> " paPndSvc=" <> tshow paPndSvc
<> " paPndQ=" <> tshow paPndQ
<> " paWorkers=" <> tshow paWorkers
<> " loadedQ=" <> tshow (loadedQueueCount lc)
<> " loadedNtf=" <> tshow (loadedNotifierCount lc)
<> " ntfLocks=" <> tshow (notifierLockCount lc)
```
Note: `smpSubs.subsCount` (queueSubscribers size) and `smpSubs.subServicesCount` (serviceSubscribers size) are **already logged** in Prometheus (lines 475-496). The log above adds all other candidate structures plus GHC RTS memory stats.
This produces a single log line every 5 minutes:
```
[INFO] MEMORY rts_live=10737418240 rts_heap=23488102400 rts_gc=4521 clients=21927 smpQSubs=1847233 smpSSubs=42 ntfQSubs=982112 ntfSSubs=31 smpPending=0 ntfPending=0 ntfStore=512844 paClients=12 paSessions=12 paActSvc=0 paActQ=0 paPndSvc=0 paPndQ=0 paWorkers=3 loadedQ=0 loadedNtf=0 ntfLocks=0
```
### What Each Metric Tells Us
| Metric | What it reveals | If growing = suspect |
|--------|----------------|---------------------|
| `rts_live` | Actual live data after last major GC | Baseline — everything else should add up to this |
| `rts_heap` | Total heap (should be ~2× rts_live) | If >> 2× live, fragmentation issue |
| `clients` | Connected client count | Known: ~22K |
| `smpQSubs` | SubscribedClients map size (queue subs) | If >> clients × avg_subs, entries not cleaned |
| `smpSSubs` | SubscribedClients map size (service subs) | Should be small |
| `ntfQSubs` | NTF SubscribedClients map (queue subs) | Same concern as smpQSubs |
| `ntfSSubs` | NTF SubscribedClients map (service subs) | Should be small |
| `smpPending` / `ntfPending` | Pending END/DELD events per client | If large, subscriber thread lagging |
| `ntfStore` | NotifierId count in NtfStore | If growing monotonically, zombie entries |
| `paClients` | Proxy connections to other servers | Should be <= unique dest servers |
| `paSessions` | Active proxy sessions | Should match paClients |
| `paActSvc` / `paActQ` | Proxy active subscriptions | If growing, entries never removed |
| `paPndSvc` / `paPndQ` | Proxy pending subscriptions | If growing, resubscription stuck |
| `paWorkers` | Active reconnect workers | If growing, workers stuck in retry |
| `loadedQ` | Cached queues in store (0 with useCache=False) | Should be 0 |
| `ntfLocks` | Notifier locks in store | Should be 0 with useCache=False |
### Interpretation Guide
**If `smpQSubs` is in the millions**: SubscribedClients is the primary leak. Entries accumulate for every queue ever subscribed to.
**If `ntfStore` grows monotonically**: Zombie notification entries (empty lists after expiration). Fix: `deleteExpiredNtfs` should remove entries with empty lists.
**If `paActSvc` + `paActQ` grow**: Proxy agent subscription maps are the leak. Fix: add cleanup when no active/pending subs exist for a server.
**If `rts_live` is much smaller than `rts_heap`**: GHC heap fragmentation. Fix: tune `-F` flag (GC trigger factor) or use `-c` (compacting GC).
**If `rts_live` ~ 10-12GB**: The live data is genuinely large. Look at which metric is the largest contributor.
**If nothing above is large but `rts_live` is large**: The leak is in a structure not measured here — likely TLS connection buffers, ByteString retention from Postgres queries, or GHC runtime overhead. Next step would be heap profiling with `-hT`.
+105 -21
View File
@@ -2,10 +2,6 @@
set -eu
# Links to scripts/configs
bin="https://github.com/simplex-chat/simplexmq/releases/latest/download"
bin_smp="$bin/smp-server-ubuntu-20_04-x86-64"
bin_xftp="$bin/xftp-server-ubuntu-20_04-x86-64"
scripts="https://raw.githubusercontent.com/simplex-chat/simplexmq/stable/scripts/main"
scripts_systemd_smp="$scripts/smp-server.service"
scripts_systemd_xftp="$scripts/xftp-server.service"
@@ -26,6 +22,8 @@ path_conf_var="/var/opt"
path_conf_smp="$path_conf_etc/simplex $path_conf_var/simplex"
path_conf_xftp="$path_conf_etc/simplex-xftp $path_conf_var/simplex-xftp /srv/xftp"
path_conf_info="$path_conf_etc/simplex-info"
path_systemd="/etc/systemd/system"
path_systemd_smp="$path_systemd/smp-server.service"
path_systemd_xftp="$path_systemd/xftp-server.service"
@@ -56,7 +54,7 @@ ${GRN}1.${NC} Install latest binaries from GitHub releases:
${GRN}2.${NC} Create server directories:
- smp: ${YLW}${path_conf_smp}${NC}
- xftp: ${YLW}${path_conf_xftp}${NC}
${GRN}3.${NC} Setup user for each server:
${GRN}3.${NC} Setup user for server:
- xmp: ${YLW}${user_smp}${NC}
- xftp: ${YLW}${user_xftp}${NC}
${GRN}4.${NC} Create systemd services:
@@ -65,7 +63,12 @@ ${GRN}4.${NC} Create systemd services:
${GRN}5.${NC} Install stopscript (systemd), update and uninstallation script:
- all: ${YLW}${path_bin_update}${NC}, ${YLW}${path_bin_uninstall}${NC}, ${YLW}${path_bin_stopscript}${NC}
Press ${GRN}ENTER${NC} to continue or ${RED}Ctrl+C${NC} to cancel installation"
Press:
- ${GRN}1${NC} to install smp server
- ${GRN}2${NC} to install xftp server
- ${RED}Ctrl+C${NC} to cancel installation
Selection: "
end="Installtion is complete!
@@ -76,27 +79,79 @@ Please checkout our server guides:
To uninstall with full clean-up, simply run: ${YLW}sudo /usr/local/bin/simplex-servers-uninstall${NC}
"
set_version() {
ver="${VER:-latest}"
case "$ver" in
latest)
bin="https://github.com/simplex-chat/simplexmq/releases/latest/download"
remote_version="$(curl --proto '=https' --tlsv1.2 -sSf -L https://api.github.com/repos/simplex-chat/simplexmq/releases/latest | grep -i "tag_name" | awk -F \" '{print $4}')"
;;
*)
bin="https://github.com/simplex-chat/simplexmq/releases/download/${ver}"
remote_version="${ver}"
;;
esac
}
os_test() {
. /etc/os-release
case "$VERSION_ID" in
20.04|22.04) : ;;
24.04) VERSION_ID='22.04' ;;
*) printf "${RED}Unsupported Ubuntu version!${NC}\nPlease file Github issue with request to support Ubuntu %s: https://github.com/simplex-chat/simplexmq/issues/new\n" "$VERSION_ID" && exit 1 ;;
esac
version="$(printf '%s' "$VERSION_ID" | tr '.' '_')"
arch="$(uname -p)"
case "$arch" in
x86_64) arch="$(printf '%s' "$arch" | tr '_' '-')" ;;
*) printf "${RED}Unsupported architecture!${NC}\nPlease file Github issue with request to support %s architecture: https://github.com/simplex-chat/simplexmq/issues/new" "$arch" && exit 1 ;;
esac
bin_smp="$bin/smp-server-ubuntu-${version}-${arch}"
bin_xftp="$bin/xftp-server-ubuntu-${version}-${arch}"
}
setup_bins() {
curl --proto '=https' --tlsv1.2 -sSf -L "$bin_smp" -o "$path_bin_smp" && chmod +x "$path_bin_smp"
curl --proto '=https' --tlsv1.2 -sSf -L "$bin_xftp" -o "$path_bin_xftp" && chmod +x "$path_bin_xftp"
eval "bin=\$bin_${1}"
eval "path=\$path_bin_${1}"
curl --proto '=https' --tlsv1.2 -sSf -L "$bin" -o "$path" && chmod +x "$path"
unset bin path
}
setup_users() {
useradd -M "$user_smp" 2> /dev/null || true
useradd -M "$user_xftp" 2> /dev/null || true
eval "user=\$user_${1}"
useradd -M "$user" 2> /dev/null || true
unset user
}
setup_dirs() {
# Unquoted varibles, so field splitting can occur
mkdir -p $path_conf_smp
chown "$user_smp":"$user_smp" $path_conf_smp
mkdir -p $path_conf_xftp
chown "$user_xftp":"$user_xftp" $path_conf_xftp
eval "path_conf=\$path_conf_${1}"
eval "user=\$user_${1}"
mkdir -p $path_conf
mkdir -p $path_conf_info
printf "local_version_%s='%s'\n" "$1" "$remote_version" >> "$path_conf_info/release"
chown -R "$user":"$user" $path_conf
unset path_conf user
}
setup_systemd() {
curl --proto '=https' --tlsv1.2 -sSf -L "$scripts_systemd_smp" -o "$path_systemd_smp"
curl --proto '=https' --tlsv1.2 -sSf -L "$scripts_systemd_xftp" -o "$path_systemd_xftp"
eval "scripts_systemd=\$scripts_systemd_${1}"
eval "path_systemd=\$path_systemd_${1}"
curl --proto '=https' --tlsv1.2 -sSf -L "$scripts_systemd" -o "$path_systemd"
unset scripts_systemd path_systemd
}
setup_scripts() {
@@ -110,32 +165,61 @@ checks() {
printf "This script is intended to be run with root privileges. Please re-run script using sudo."
exit 1
fi
set_version
os_test
mkdir -p $path_conf_info
}
main() {
checks
printf "%b\n%b\n" "${BLU}$logo${NC}" "$welcome"
printf "%b\n%b" "${BLU}$logo${NC}" "$welcome"
read ans
case "$ans" in
1) setup='smp' ;;
2) setup='xftp' ;;
*) printf 'Installation aborted.\n' && exit 0 ;;
esac
printf "Installing binaries..."
setup_bins
for i in $setup; do
setup_bins "$i"
done
printf "${GRN} Done!${NC}\n"
printf "Creating users..."
setup_users
for i in $setup; do
setup_users "$i"
done
printf "${GRN} Done!${NC}\n"
printf "Creating directories..."
setup_dirs
for i in $setup; do
setup_dirs "$i"
done
printf "${GRN} Done!${NC}\n"
printf "Creating systemd services..."
setup_systemd
for i in $setup; do
setup_systemd "$i"
done
printf "${GRN} Done!${NC}\n"
printf "Installing stopscript, update and uninstallation script..."
setup_scripts
printf "${GRN} Done!${NC}\n"
printf "%b" "$end"
-181
View File
@@ -1,181 +0,0 @@
name: simplexmq
version: 5.6.1.0
synopsis: SimpleXMQ message broker
description: |
This package includes <./docs/Simplex-Messaging-Server.html server>,
<./docs/Simplex-Messaging-Client.html client> and
<./docs/Simplex-Messaging-Agent.html agent> for SMP protocols:
.
* <https://github.com/simplex-chat/simplexmq/blob/master/protocol/simplex-messaging.md SMP protocol>
* <https://github.com/simplex-chat/simplexmq/blob/master/protocol/agent-protocol.md SMP agent protocol>
.
See <https://github.com/simplex-chat/simplex-chat terminal chat prototype> built with SimpleXMQ broker.
homepage: https://github.com/simplex-chat/simplexmq#readme
license: AGPL-3
author: simplex.chat
maintainer: chat@simplex.chat
copyright: 2020-2022 simplex.chat
category: Chat, Network, Web, System, Cryptography
extra-source-files:
- README.md
- CHANGELOG.md
- cbits/sha512.h
- cbits/sntrup761.h
dependencies:
- aeson == 2.2.*
- ansi-terminal >= 0.10 && < 0.12
- asn1-encoding == 0.9.*
- asn1-types == 0.3.*
- async == 2.2.*
- attoparsec == 0.14.*
- base >= 4.14 && < 5
- base64 == 1.0.*
- case-insensitive == 1.2.*
- composition == 1.0.*
- constraints >= 0.12 && < 0.14
- containers == 0.6.*
- crypton == 0.34.*
- crypton-x509 == 1.7.*
- crypton-x509-store == 1.6.*
- crypton-x509-validation == 1.6.*
- cryptostore == 0.3.*
- data-default == 0.7.*
- direct-sqlcipher == 2.3.*
- directory == 1.3.*
- filepath == 1.4.*
- hourglass == 0.2.*
- http-types == 0.12.*
- http2 >= 4.2.2 && < 4.3
- ini == 0.4.1
- iproute == 1.7.*
- iso8601-time == 0.1.*
- memory == 0.18.*
- mtl >= 2.3.1 && < 3.0
- network >= 3.1.2.7 && < 3.2
- network-info >= 0.2 && < 0.3
- network-transport == 0.5.6
- network-udp >= 0.0 && < 0.1
- optparse-applicative >= 0.15 && < 0.17
- process == 1.6.*
- random >= 1.1 && < 1.3
- simple-logger == 0.1.*
- socks == 0.6.*
- sqlcipher-simple == 0.4.*
- stm == 2.5.*
- temporary == 1.3.*
- time == 1.12.*
- time-manager == 0.0.*
- tls >= 1.7.0 && < 1.8
- transformers == 0.6.*
- unliftio == 0.2.*
- unliftio-core == 0.2.*
- websockets == 0.12.*
- yaml == 0.11.*
- zstd == 0.1.3.*
flags:
swift:
description: Enable swift JSON format
manual: True
default: False
use_crypton:
description: Use crypton etc. in cryptostore
manual: True
default: True
when:
- condition: flag(swift)
cpp-options:
- -DswiftJSON
- condition: impl(ghc >= 9.6.2)
dependencies:
- bytestring == 0.11.*
- template-haskell == 2.20.*
- text >= 2.0.1 && < 2.2
- condition: impl(ghc < 9.6.2)
dependencies:
- bytestring == 0.10.*
- template-haskell == 2.16.*
- text >= 1.2.3.0 && < 1.3
library:
source-dirs: src
c-sources:
- cbits/sha512.c
- cbits/sntrup761.c
include-dirs: cbits
extra-libraries: crypto
executables:
smp-server:
source-dirs: apps/smp-server
main: Main.hs
dependencies:
- simplexmq
ghc-options:
- -threaded
- -rtsopts
ntf-server:
source-dirs: apps/ntf-server
main: Main.hs
dependencies:
- simplexmq
ghc-options:
- -threaded
- -rtsopts
xftp-server:
source-dirs: apps/xftp-server
main: Main.hs
dependencies:
- simplexmq
ghc-options:
- -threaded
- -rtsopts
smp-agent:
source-dirs: apps/smp-agent
main: Main.hs
dependencies:
- simplexmq
ghc-options:
- -threaded
- -rtsopts
xftp:
source-dirs: apps/xftp
main: Main.hs
dependencies:
- simplexmq
ghc-options:
- -threaded
- -rtsopts
tests:
simplexmq-test:
source-dirs: tests
main: Test.hs
dependencies:
- simplexmq
- deepseq == 1.4.*
- generic-random == 1.5.*
- hspec == 2.11.*
- hspec-core == 2.11.*
- HUnit == 1.6.*
- QuickCheck == 2.14.*
- silently == 1.2.*
- main-tester == 0.2.*
- timeit == 2.0.*
ghc-options:
# - -haddock
- -Wall
- -Wcompat
- -Werror=incomplete-patterns
- -Wredundant-constraints
- -Wincomplete-record-updates
- -Wincomplete-uni-patterns
- -Wunused-type-patterns
@@ -0,0 +1,180 @@
# SMP router message storage
## Problem
Currently SMP routers store all queues in router memory. As the traffic grows, so does the number of undelivered messages. What is worse, Haskell is not avoiding heap fragmentation when messages are allocated and then de-allocated - undelivered messages use ByteString and GC cannot move them around, as they use pinned memory.
## Possible solutions
### Solution 1: solve only GC fragmentation problem
Move from ByteString to some other primitive to store messages in memory long term, e.g. ShortByteString, or manage allocation/de-allocation of stored messages manually in some other way.
Pros: the simplest solution that avoids substantial re-engineering of the router.
Cons:
- not a long term solution, as memory growth still has limits.
- may be ineffective, as it introduces additional copying of bytes.
### Solution 2: move message storage to hard drive
Use files or RocksDB to store messages.
Pros:
- much lower memory usage.
- no message loss in case of abnormal router termination (important until clients have delivery redundancy).
- this is a long term solution, and at some point it might need to be done anyway.
Cons:
- substantial re-engineering costs and risks.
- metadata privacy. Currently we only save undelivered messages when router is restarted, with this approach all messages will be stored for some time. this argument is limited, as hosting providers of VMs can make memory snapshots too, on the other hand they are harder to analyze than files. On another hand, with this approach messages will be stored for a shorter time.
#### RocksDB and other key-value stores
The downside of any key-value stores is that they don't seem to have efficient primitives for sequential delivery. While sequential delivery can be modelled with linked lists, they would require 1 key insert (on send), 3 key updates (1 update to update queue data on send, 1 update of the last message to point to the next, 1 update on delivery or message expiration) and 1 key deletion (on delivery or message expiration) for each delivered message.
This might result in substantial write amplification and compacting costs.
In general, tree structures that are efficient for quick lookups and updates, given approximately fixed value size, are inefficient for modelling queues.
#### Files
The upside of files is that they are well suited for sequential delivery and don't result in the same churn, with careful design, as trees do.
The downside of filesystem is that it does not scale well with the large number of files in a folder, so queues will have to be spread across multiple folders, following tree-like structure.
I could not find an available library that efficiently models sequential delivery in highly concurrent environment.
A possible design could be the following.
##### Queue folder and files
Each message queue is stored in its own folder (see below on folder locations). Folder would contain these files:
- `messages.abcd.log` - the file that is used to read messages from, sequentially
- `messages.efgh.log` - the optional file that is used to write messages, in case it is different from read file.
- `queue.log` - append-only file where the last line represents the current queue state
- `queue.timestamp.log` - previous states of queue.log file
Each line in "queue.log" file has this syntax
```abnf
queueLogLine =
%s"read_file=" base64
%s"read_msg=" digits
%s"read_byte=" digits
%s"write_file=" base64
%s"write_msg=" digits
```
When queue is first requested by the router:
```c
if queue folder exists:
read queue state from last line of queue.log
if queue.log contained more than one line: // compaction
copy queue.log to queue.timestamp.log
write one line queue state to queue.log
else:
create queue folder
create messages.abcd.log (abcd is some random string)
read_msg = 0
read_byte = 0
create queue.log with one line: "read_file=abcd read_msg=0 read_byte=0 write_files=abcd write_msg=0"
open read_file in ReadMode and seek to read_byte position
nextReadByte = read_byte
nextReadMsg = read_msg
open write_file in AppendMode
```
When message is added to the queue (assumes that queue state is loaded to router memory, if not the previous section will be done first):
```c
if write_msg > max_queue_messages:
return quota error
else if write_msg = max_queue_messages:
add quota_exceeded message to write_file
update queue state: write_msg += 1
append updated queue state to queue.log
else
// It is required that `max_queue_messages < max_file_messages`,
// so that we never need more than one additional write file.
if write_msg >= max_file_messages: // queue file rotation
create messages.efgh.log // efgh is some random string
update queue state: write_file=efgh write_msg=0 // read file remains the same as it was
append updated queue state to queue.log
copy queue.log to queue.timestamp.log
// `old` needs to be defined to limit the number and storage duration,
// preserving not more than N files, and not more than M days files, "and then some"
// (that is if the queue has high churn, we have file from M days before in any case, for any debugging).
delete `old` `queue.timestamp.log` files
write one line queue state to queue.log // compaction
add message to write_file
update queue state: write_msg += 1
append updated queue state to queue.log
```
The above algorithm assumes `max_queue_messages < than max_file_messages`, so that we never need more than one write file.
When message is delivered, it is simply read from the read queue, queue state does not change yet:
```c
if nextReadMsg > read_msg:
deliver cached message, no need to read it again
else
read message from read_file handle
nextReadMsg = read_msg + 1
nextReadByte = current position in file
```
When message delivery is acknowledged, the read queue needs to be advanced, and possibly switched to read from the current write queue:
```c
if nextReadByte == read_byte:
return error // nothing was delivered
else if nextReadByte = EOF:
// end of file is reached, possibly some other condition,
// but it should allow changing max_file_messages on server restart
currReadFile = read_file
read_file = write_file
read_msg = 0
read_byte = 0
append updated queue state to queue.log
delete currReadFile
else
read_msg += 1
read_byte = nextReadByte
// `seek` should not be necessary, as the handle is already at nextReadByte position here
// seek to read_byte
append updated queue state to queue.log
```
The above algorithm delegates the problem of compaction and fragmentation management to file system, that is very optimized for such scenarios.
Also, read and write files will grow to almost a constant size, so the space they used may be re-used.
An important consideration is that writes to queue.log and message.log files and queue state modifications have to be sequential, without concurrency - it can be managed with the usual locks.
##### Queue folders structure
Most Linux systems use EXT4 filesystem where the file lookup time scales linearly to the number of files. While alternatives with logarithmic lookup time exist (XFS), they may be very complex to configure on the existing systems.
So storing all queue folders in one folder won't scale.
To solve this problem we could use recipient queue ID in base64url format not as a folder name, but as a folder path, splitting it to path fragments of some length. The number of fragments can be configurable and migration to a different fragment size can be supported as the number of queues on a given router grows.
Currently, queue ID is 24 bytes random number, thus allowing 2^192 possible queue IDs. If we assume that a router must hold 1b queues, it means that we have ~2^162 possible addresses for each existing queue. 24 bytes in base64 is 32 characters that can be split into say 8 fragments with 4 characters each, so that queue folder path for queue with ID `abcdefghijklmnopqrstuvwxyz012345` would be:
`/var/opt/simplex/messages/abcd/efgh/ijkl/mnop/qrst/uvwx/yz01/2345`
The maximum theoretic number of the folders on the 1st level is 64^4, or 2^24 ~ 16m - this is probably still a large number of subfolders for EXT4. Given that addresses are random, all the possible combinations in the first folder can be used with a large number of queues.
So we could use an unequal split of path, two letters each and the last being long:
`/var/opt/simplex/messages/ab/cd/ef/ghijklmnopqrstuvwxyz012345`
The first three levels in this case can have 4096 subfolders each, and it gives 68b possible subfolders (64^2^3), so the last level will be sparse in case of 1b queues on the router. So we could make it 4 levels with 2 letters to never think about it, accounting for a large variance of the random numbers distribution:
`/var/opt/simplex/messages/ab/cd/ef/gh/ijklmnopqrstuvwxyz012345`
+162
View File
@@ -0,0 +1,162 @@
# Sharing protocol ports with HTTPS
Some networks block all ports other than web ports, including port 5223 used for SMP protocol by default. Running SMP routers on a common web port 443 would allow them to work on more networks. The routers would need to provide an HTTPS page for browsers (and probes).
## Problem
Browsers and tools rely on system CA bundles instead of certificate pinning.
The crypto parameters used by HTTPS are different from what the protocols use.
Public certificate providers like LetsEncrypt can only sign specific types of keys and Ed25519 isn't one of them.
This means a router should distinguish browser and protocol clients and adjust its behavior to match.
## Solution
`tls` package has a server hook that allows producing a different set of `TLS.Credentials` according to a client-provided "Server Name Indication" extension.
Since LE certificates are only handed out to domain names, TLS client will be sending the SNI.
However client transports are constructed over connected sockets and the SNI wouldn't be present unless explicitly requested.
When a client sends SNI, then it's a browser and web credentials should be used.
Otherwise it's a protocol client to be offered the self-signed ca, cert and key.
When a transport colocated with a HTTPS, its ALPN list should be extended with `h2 http/1.1`.
The browsers will send it, and it should be checked before running transport client.
If HTTP ALPN is detected, then the client connection is served with HTTP `Application` instead (the same "router information" page).
If some client connects to router IP, doesn't send SNI and doesn't send ALPN, it will look like a pre-handshake client.
In that case a router will send its handshake first.
This can be mitigated by delaying its handshake and letting the probe to issue its HTTP request.
## Implementation plan
An unmodified client should be able to use protocols on port 443 right away.
The switchover happens inside `runTransportServerState` before `runClient`:
```haskell
runServer (tcpPort, ATransport t) = do
-- ...
runTransportServerState_ ss started tcpPort serverParams tCfg $ \socket h -> do -- expose raw socket for warp-tls internals to attach
negotiated <- getSessionALPN
if allowHTTP t && isHTTP negotiated -- only attempt the switch for the TLS transport
then runHTTP socket (tlsContext h)-- ... collect data and produce values needed to run WAI Application
else runClient serverSignKey t h `runReaderT` env -- performs serverHandshake etc as usual
```
The web app and router live outside, so `runHttp` has to be provided by the `runSMPServer` caller.
Additonally, Warp is using its `InternalInfo` object that's scoped to `withII` bracket.
```haskell
runServer ini = do
-- ...
runWebServer ini ServerInformation {config, information} $ if sharedHttps then Nothing else webHttpsParams -- suppress serving https
if sharedHttps
then withRunHTTP staticFilesPath \attachStatic -> runSMPServer cfg (Just attachStatic) -- provide wrapped application runner
else runSMPServer cfg Nothing
```
### Upstream
The implementation relies on a few modification to upstream code:
- `warp-tls`: The library provides `httpOverTls`, but it wants to do handshake itself.
Since we have to do the handshake to switch on ALPN, the setup function has to be split.
This is a resonable change that may be upstreamed and nothing blocks us from using the recent version.
- `warp`: Only the re-export of `serveConnection` is needed.
Unfortunately the most recent `warp` version can't be used right away due to dependency cascade around `http-5` and `auto-update-2`.
So a fork containing the backported re-export has to be used until the dependencies are refreshed.
### TLS.ServerParams
When a router has port sharing enabled, a new set of TLS params is loaded and combined with transport params:
```haskell
newEnv config = do
-- ...
tlsServerParams <- loadTLSServerParams caCertificateFile certificateFile privateKeyFile (alpn transportConfig)
sharedServerParams <- forM ((,) <$> sharedHttpsCredentials config <*> alpn transportConfig) $ \((chain, key), alpn) ->
let ca = Nothing -- It is possible to provide CA certificate, but it is typical for web server to use combined certificate chains
loadHTTPSServerParams tlsServerParams ca chain key alpn
```
`loadHTTPSServerParams` extends params with:
1. `onALPNClientSuggest` hook gets `["h2", "http/1.1"]` added to the ALPN list which is now required.
2. `onServerNameIndication` hook added, which upon detecting client SNI prepends the web credentials.
3. `sharedCredentials = T.Credentials []` should be done to prevent transport credentials confusing browsers.
But that aborts key exchange somewhere in tls internals, so disabled for now.
As a workaround, another set of dummy credentials can be provided in the hope that any sane browser would reject them.
Like, RC4 ciphers, "impossible" digest combination, etc.
### supportedParameters
TLS certificate chains provided by LetsEncrypt use ECDSA/P256 and that requires extending `supportedParameters` with things disabled in transports:
```haskell
browserCiphers =
[ TE.cipher_TLS13_AES128CCM8_SHA256
, TE.cipher_ECDHE_ECDSA_AES128CCM8_SHA256
, TE.cipher_ECDHE_ECDSA_AES256CCM8_SHA256
]
browserGroups =
[ T.P256
]
browserSigs =
[ (T.HashSHA256, T.SignatureECDSA),
(T.HashSHA384, T.SignatureECDSA)
]
```
This may not be enough for other certificate providers.
## Configuration
> XXX: This is for the current implementation and should be updated.
Web certificate chain is picked up from the WEB section:
```ini
[TRANSPORT]
port: 443
[WEB]
https: 443
cert: /etc/opt/simplex/web.cert
key: /etc/opt/simplex/web.key
# Alternatively, with a proper access configuration, the paths can point to the LE creds directly:
# cert: /etc/letsencrypt/live/smp.hostname.tld/fullchain.pem
# key: /etc/letsencrypt/live/smp.hostname.tld/privkey.pem
```
When `TRANSPORT.port` matches `WEB.https` the transport router becomes shared.
Perhaps a more desirable option would be explicit configuration resulting in additional transported to run:
```ini
[TRANSPORT]
port: 5223 ; pure protocol transport
# control_port: 5224
shared_port: 443 ; variant 1: register in TRANSPORT
[WEB]
https: 443
cert: /etc/opt/simplex/web.cert
key: /etc/opt/simplex/web.key
# transport: on ; variant 2:
```
## Caveats
Serving static files and the protocols together may pose a problem for those who currently use dedicated web servers as they should switch to embedded http handlers.
As before, using embedded HTTP server is increasing attack surface.
Users who want to run everything on a single host will have to add an extra IP address and bind routers to specific IPs instead of 0.0.0.0.
An amalgamated router binary can be provided that would contain both SMP and XFTP routers, where transport will dispatch connections by handshake ALPN.
## Alternative: Use transports routable with reverse-proxies
An "industrial" reverse proxy may do the ALPN routing, serving HTTP by itself and delegating `smp` and `xftp` to protocol servers.
Same with the `websockets`.
Since this in effect does TLS termination, the protocol routers will have to rely on credentials from protocol handshakes.
@@ -0,0 +1,16 @@
# Expiring messages in journal storage
## Problem
The journal storage routers recently migrated to do not delete delivered or expired messages, they only update pointers to journal file lines. The messages are actually deleted when the whole journal file is deleted (when fully deleted or fully expired).
The problem is that in case the queue stops receiving the new messages then writing of messages won't switch to the new journal file, and the current journal file containing delivered or expired messages would never be deleted.
## Solution
Remove current journal file and update queue_state.log during message expiration of "idle" queue (that is, without any new messages received or delivered within 3 hours) in case when:
- the queue is "empty" after the expiration
- the queue contains only quota marker(s), in which case move them to a new journal file and update the queue_state accordingly. Quota markers can be kept indefinitely to prevent writing the new messages to the dormant queues that reached capacity, so it's important to handle this case.
Also remove current journal file when the queue is opened in case it is empty (as it would not be ever expired in case it remains empty), and also update queue_state.log
+416
View File
@@ -0,0 +1,416 @@
# Fix subQ deadlock: blocking writeTBQueue inside connLock
## Problem
Users report that message reception silently and permanently stops across all connections, with no error alerts. The app appears functional but no messages arrive. Recovery requires restart.
Root cause: a deadlock between worker threads holding `connLock` and the `agentSubscriber` (sole `subQ` reader).
### The deadlock mechanism
`subQ` (`TBQueue ATransmission`, capacity 4096 on mobile / 1024 on desktop) is the single pipeline between the agent layer and the chat layer. The `agentSubscriber` thread (`Commands.hs:4373`) is its **sole reader**.
Three code sites hold `connLock` and call blocking `writeTBQueue subQ` without a fullness check. When `subQ` is full, these block while holding the lock. If `agentSubscriber` simultaneously needs the same `connLock` (via `sendMessagesB_``withConnLocks`), it blocks too — creating a circular wait:
- **Worker**: holds `connLock(X)`, waits for `subQ` space (needs `agentSubscriber` to read)
- **agentSubscriber**: sole `subQ` reader, waits for `connLock(X)` (needs worker to release)
- **Result**: permanent silent deadlock — no exception, no alert, all connections blocked
### Confirmed deadlock scenarios
**Scenario 1**: Delivery worker during queue rotation test
```
Delivery worker: agentSubscriber (sole subQ reader):
withConnLock(X) [2187] readTBQueue subQ → processAgentMessageConn
...DB operations... → sendPendingGroupMessages (on CON/SENT/QCONT)
notify → writeTBQueue subQ [2238] → batchSendConnMessages → deliverMessagesB
[BLOCKED — subQ full] → withAgent sendMessagesB [synchronous]
→ sendMessagesB_ → withConnLocks({..X..}) [1708]
[BLOCKED — connLock(X) held]
```
**Scenario 2**: Async command worker during message ACK with notification
```
Async cmd worker: agentSubscriber (sole subQ reader):
tryWithLock "ICAck" [1930→1824] readTBQueue subQ → processAgentMessageConn
→ withConnLock(X) → sendPendingGroupMessages
→ ack → ackQueueMessage [1899] → sendMessagesB_ → withConnLocks({..X..})
→ sendMsgNtf [2381] [BLOCKED — connLock(X) held]
→ writeTBQueue subQ [2386]
[BLOCKED — subQ full]
```
**Scenario 3**: Synchronous `ackMessage'` API (same mechanism as Scenario 2 but from external API caller)
```
ackMessage' caller: agentSubscriber (sole subQ reader):
withConnLock(X) [2254] → sendMessagesB_ → withConnLocks({..X..})
→ ack → ackQueueMessage [2267] [BLOCKED — connLock(X) held]
→ sendMsgNtf [2381]
→ writeTBQueue subQ [2386]
[BLOCKED — subQ full]
```
### ConnId overlap verified
No guard prevents a connection undergoing queue rotation (AM_QTEST_) or ACK processing from being included in `sendMessagesB_`'s batch. During these operations, the connection has `connStatus == ConnReady`, passing all filters in `memberSendAction`.
### Cascade amplification
Once any single deadlock triggers, `subQ` never drains. ALL other threads that attempt `writeTBQueue subQ` block progressively — their locks are held forever too. The entire threading system freezes within seconds.
### Affected code sites (blocking `writeTBQueue subQ` inside `connLock`)
| Site | File | Lock line | Write line | Events written |
|------|------|-----------|------------|----------------|
| `runSmpQueueMsgDelivery::notify` | Agent.hs | 2187 | 2238 | SWITCH SPCompleted, ERR INTERNAL |
| `runSmpQueueMsgDelivery::internalErr/notifyDel` | Agent.hs | 2187 | 2238 (via notifyDel→notify) | ERR INTERNAL + delMsg |
| `ackQueueMessage::sendMsgNtf` | Agent.hs | 2254 or 1930 | 2386 | MSGNTF |
### Safe patterns that already exist in the codebase
1. **`isFullTBQueue` + pending TVar** (used at `runCommandProcessing` lines 1782-1784/1937, and `runProcessSMP` lines 3027-3029/3216):
```haskell
-- Before processing (e.g. line 1782):
pending <- newTVarIO []
-- During processing — safe notify (e.g. line 1937):
notify cmd =
let t = (corrId, connId, AEvt (sAEntity @e) cmd)
in atomically $ ifM (isFullTBQueue subQ) (modifyTVar' pendingCmds (t :)) (writeTBQueue subQ t)
-- After processing — flush (e.g. line 1784):
mapM_ (atomically . writeTBQueue subQ) . reverse =<< readTVarIO pending
```
2. **`nonBlockingWriteTBQueue`** (used at Client.hs:789, NtfSubSupervisor.hs:507):
```haskell
nonBlockingWriteTBQueue q x = do
sent <- atomically $ tryWriteTBQueue q x
unless sent $ void $ forkIO $ atomically $ writeTBQueue q x
```
Note: `nonBlockingWriteTBQueue` does NOT preserve ordering — the spawned background thread may complete out of order relative to subsequent direct writes from the same calling thread.
### Exhaustive proof: no other deadlock scenarios exist
All 15 `withConnLock` sites in Agent.hs were analyzed. Only 3 write to `subQ`:
| withConnLock site | Writes subQ? | Safe? |
|-------------------|-------------|-------|
| switchConnectionAsync' (899) | No | ✓ |
| setConnShortLinkAsync' (995) | No | ✓ |
| setConnShortLink' (1031) | No | ✓ |
| deleteConnShortLink' (1075) | No | ✓ |
| allowConnection' (1407) | No | ✓ |
| acceptContact' (1417) | No | ✓ |
| sendMessagesB_ (1708, `withConnLocks`) | No | ✓ |
| tryWithLock/runSmpCommand (1930) | Yes (1937) | ✓ — `isFullTBQueue` check |
| tryMoveableWithLock/runSmpCommand (1931) | Yes (1937) | ✓ — `isFullTBQueue` check |
| **runSmpQueueMsgDelivery AM_QTEST_ (2187)** | **Yes (2238)** | **✗ — DEADLOCK** |
| **ackMessage' (2254)** | **Yes (2386)** | **✗ — DEADLOCK** |
| switchConnection' (2298) | No | ✓ |
| abortConnectionSwitch' (2328) | No | ✓ |
| synchronizeRatchet' (2351) | No | ✓ |
| suspendConnection' (2390) | No | ✓ |
| **processSMP (3037)** | Yes (3216) | ✓ — `isFullTBQueue` check |
Note: `processSMP` (line 3037) holds `connLock` and its local `notify` (line 3216) writes to `subQ`, but it uses the safe `isFullTBQueue` pattern. Its `ack` (line 3196) uses `enqueueCmd` (DB-only), NOT `ackQueueMessage`. The actual `ackQueueMessage` runs later from the async command worker via ICAck/ICAckDel.
Other lock pairs checked — no circular dependencies:
- `connLock × DB MVar`: DB never acquires connLock
- `entityLock × connLock`: consistent ordering (entity first in chat, conn in agent)
- `connLock(X) × connLock(Y)`: single agentSubscriber thread, one `withConnLocks` at a time
---
## Deadlock call graph: agentSubscriber → connLock
All deadlock paths require `agentSubscriber` to synchronously acquire `connLock`. Exhaustive analysis shows that **every such path converges on a single agent function**: `sendMessagesB_``withConnLocks` (Agent.hs:1708). No other agent API function called synchronously from the agentSubscriber acquires connLock.
Verified (FACT): `ackMessageAsync``enqueueCommand` only (no connLock). `toggleConnectionNtfs` → no lock. `deleteConnectionAsync``deleteLock` not `connLock`. `joinConnectionAsync``withInvLock` not `connLock`.
Also verified (FACT): `Lock = TMVar Text` (Lock.hs:24) is **non-reentrant** — double acquisition on the same thread deadlocks.
### All 22 trigger paths
Every path goes through `deliverMessage`/`deliverMessages`/`deliverMessagesB``withAgent sendMessagesB``sendMessagesB_``withConnLocks`:
| # | Trigger | Chat function | ConnIds locked | Risk |
|---|---------|--------------|----------------|------|
| 1 | Group CON (Invitee) | `introduceToAll` → broadcast XGrpMemNew | **ALL member connIds** | **HIGHEST** |
| 2 | Group MSG XGrpLinkAcpt | `introduceToRemaining` → broadcast | **ALL member connIds** | **HIGHEST** |
| 3 | Group CON (Invitee) | `sendIntroductions` → batch intros to new member | new member connId | Medium |
| 4 | Group CON (Invitee) | `sendHistory` → batch to new member | new member connId | Medium |
| 5 | Group CON | `sendPendingGroupMessages` | member connId | Medium |
| 6 | Group SENT | `sendPendingGroupMessages` | member connId | Medium |
| 7 | Group QCONT | `sendPendingGroupMessages` | member connId | Medium |
| 8 | Group CON (PendingReview) | `introduceToModerators` → to moderators | moderator connIds | Medium |
| 9 | Group CON (PreMember) | `sendXGrpMemCon` → to host | host connId | Low |
| 10 | Group CON (PreMember) | `probeMatchingMemberContact` → probes + hashes | member + N matching connIds | Medium |
| 11 | Direct CON | `probeMatchingMembers` → probes + hashes | contact + N matching connIds | Medium |
| 12 | Direct JOINED | `sendAutoReply` | contact connId | Low |
| 13 | Group JOINED | `sendGroupAutoReply` | member connId | Low |
| 14 | Group INV | `sendXGrpMemInv` → to host | host connId | Low |
| 15 | Group INV (legacy) | `sendGrpInvitation` → to contact | contact connId | Low |
| 16 | Group MSG XGrpMemInv | `xGrpMemInv``sendGroupMemberMessage` | re-member connId | Low |
| 17 | Group MSG XGrpMemDel | `forwardToMember` | deleted member connId | Low |
| 18 | Group MSG XGrpLinkMem | `probeMatchingMemberContact` | member + N matching connIds | Medium |
| 19 | Group MSG (dup relay) | `saveGroupRcvMsg` error → `sendDirectMemberMessage` | forwarder connId | Low |
| 20 | SFDONE | `sendFileDescriptions` → to recipients | recipient connIds | Medium |
| 21 | Group MSG XGrpLinkAcpt | `sendHistory` → to accepted member | accepted member connId | Medium |
| 22 | Direct MSG (autoAccept) | `autoAcceptFile` → inline accept reply | contact connId | Low (test-only config) |
### Key observations
1. **Single bottleneck**: All 22 paths converge on `sendMessagesB_``withConnLocks` (Agent.hs:1708). The deadlock is between this lock acquisition and any worker thread holding `connLock` + blocking on `writeTBQueue subQ`.
2. **Highest-risk paths** (#1, #2): Broadcasting to ALL group members in `introduceToAll` / `introduceToRemaining` acquires `withConnLocks` on ALL member connIds in a single batch. For large groups, this holds the agentSubscriber thread for a long time, during which subQ fills, which causes worker threads holding connLock on any of those connIds to deadlock.
3. **Medium-risk paths** (#5-7): `sendPendingGroupMessages` fires on every CON/SENT/QCONT. These are frequent and lock the member's connId, which is the SAME connId that a delivery worker or ACK worker may hold while writing to subQ.
---
## Analysis: `withConnLocks` in `sendMessagesB_`
### FACT: the lock protects ratchet encryption state
`sendMessagesB_` (Agent.hs:1708-1713) acquires `withConnLocks` and executes:
1. **`getConn_`** — reads connection metadata, send queues from DB
2. **`setConnPQSupport`** — updates PQ encryption flag per connection
3. **`enqueueMessagesB`** → `enqueueMessageB``storeSentMsg_` which calls:
- **`updateSndIds`** (AgentStore.hs:899) — increments `internalSndId` (sequential send counter)
- **`agentRatchetEncryptHeader`** (Agent.hs:3698) — reads current ratchet via `getRatchetForUpdate`, encrypts message header via `rcEncryptHeader`, writes advanced ratchet state via `updateRatchet`
- **`createSndMsg`** + **`createSndMsgDelivery`** — inserts message and delivery records
All operations run within `unsafeWithStore``withTransaction` (single DB transaction per batch).
### FACT: the lock CANNOT be removed
Without `withConnLocks`, concurrent `sendMessagesB_` calls targeting the same connection would:
- Read the same ratchet state, both encrypt, one overwrite the other → **ratchet desync** (unrecoverable)
- Get duplicate `internalSndId` values → **message ID collision**
- Race on `setConnPQSupport` → **PQ state inconsistency**
The lock serializes ALL operations on the connection's encryption state. Removing it would introduce data corruption.
Note: `sendMessage` (singular, line 530) uses the same `sendMessagesB_` function — there is no lock-free send path.
### Eliminated strategies
- **Strategy C (remove lock)**: The lock protects ratchet encryption. Removing it causes unrecoverable ratchet desync. Eliminated.
- **Strategy A (async dispatch)**: All 22 chat-layer callers use `deliverMessagesB` return values (delivery IDs, PQ state) synchronously. `forkIO` loses results. Eliminated.
- **Strategy W (isFullTBQueue + pending TVar)**: The existing pattern (lines 1937, 3216) buffers events in a local TVar and flushes after lock release. Between lock release and flush, another thread can acquire the same connLock and write events to subQ — reordering events within the same connection. This trades a visible deadlock for invisible ordering bugs. Eliminated.
- **Strategy O (per-connection overflow queues)**: Bounded overflow queues with "drop when full" were analyzed. Drop consequences are unacceptable at 5 of 6 write sites — CONF, INFO, CON cause permanent connection failure after ACK; INV loses connection invitations; SENT/MERR leave messages stuck forever. Unbounded overflow defeats backpressure. Eliminated.
---
## Solution: move subQ writes outside connLock
### Root cause
The `writeTBQueue subQ` calls at the 3 deadlock sites are inside `connLock` by accident of code structure, not necessity. `connLock` protects ratchet encryption state and DB consistency. The `notify` calls write informational events to `subQ` — they do not modify any state that `connLock` protects.
Moving the writes outside the lock scope eliminates the deadlock: blocking `writeTBQueue subQ` without holding `connLock` is safe — agentSubscriber is free to acquire the lock, process events, and drain `subQ`.
### Why reordering doesn't matter at these sites
The chat layer handlers for the 3 deadlock site events do NOT advance the ratchet:
| Event | Chat handler | Calls sendMessagesB_? |
|-------|-------------|----------------------|
| SWITCH SPCompleted | Creates internal chat item, updates UI | **No** |
| ERR INTERNAL | Logs error to view | **No** |
| MSGNTF | `toView CEvtNtfMessage` → empty output | **No** |
Events that DO trigger ratchet advances (CON, SENT, QCONT → `sendPendingGroupMessages``sendMessagesB_`) are all already written OUTSIDE `connLock` in the current code.
Ratchet state lives in the DB, not in subQ events. agentSubscriber processes events sequentially regardless of arrival order. The SENT-before-SWITCH race already exists in the current code (new queue worker writes SENT outside connLock while old queue worker writes SWITCH inside connLock).
### Fix: Site 1 — `runSmpQueueMsgDelivery` AM_QTEST_ (line 2187)
Restructure `withConnLock` to return the event, write outside.
**Current code** (Agent.hs:2187-2214):
```haskell
AM_QTEST_ -> withConnLock c connId "runSmpQueueMsgDelivery AM_QTEST_" $ do
withStore' c $ \db -> setSndQueueStatus db sq Active
SomeConn _ conn <- withStore c (`getConn` connId)
case conn of
DuplexConnection cData' rqs sqs -> do
let addr = qAddress sq
case findQ addr sqs of
Just SndQueue {dbReplaceQueueId = Just replacedId, primary} ->
case removeQP (\sq' -> dbQId sq' == replacedId && not (sameQueue addr sq')) sqs of
Nothing -> internalErr msgId "sent QTEST: queue not found in connection"
Just (sq', sq'' : sqs') -> do
checkSQSwchStatus sq' SSSendingQTEST
atomically $ TM.delete (qAddress sq') $ smpDeliveryWorkers c
withStore' c $ \db -> do
when primary $ setSndQueuePrimary db connId sq
deletePendingMsgs db connId sq'
deleteConnSndQueue db connId sq'
let sqs'' = sq'' :| sqs'
conn' = DuplexConnection cData' rqs sqs''
cStats <- connectionStats c conn'
notify $ SWITCH QDSnd SPCompleted cStats -- DEADLOCK
_ -> internalErr msgId "sent QTEST: ..." -- DEADLOCK (via notifyDel → notify)
_ -> internalErr msgId "sent QTEST: ..." -- DEADLOCK
_ -> internalErr msgId "QTEST sent not in duplex ..." -- DEADLOCK
```
**New code:**
```haskell
AM_QTEST_ -> do
evt_ <- withConnLock c connId "runSmpQueueMsgDelivery AM_QTEST_" $ do
withStore' c $ \db -> setSndQueueStatus db sq Active
SomeConn _ conn <- withStore c (`getConn` connId)
case conn of
DuplexConnection cData' rqs sqs -> do
let addr = qAddress sq
case findQ addr sqs of
Just SndQueue {dbReplaceQueueId = Just replacedId, primary} ->
case removeQP (\sq' -> dbQId sq' == replacedId && not (sameQueue addr sq')) sqs of
Nothing -> pure $ Left "sent QTEST: queue not found in connection"
Just (sq', sq'' : sqs') -> do
checkSQSwchStatus sq' SSSendingQTEST
atomically $ TM.delete (qAddress sq') $ smpDeliveryWorkers c
withStore' c $ \db -> do
when primary $ setSndQueuePrimary db connId sq
deletePendingMsgs db connId sq'
deleteConnSndQueue db connId sq'
let sqs'' = sq'' :| sqs'
conn' = DuplexConnection cData' rqs sqs''
cStats <- connectionStats c conn'
pure $ Right $ SWITCH QDSnd SPCompleted cStats
_ -> pure $ Left "sent QTEST: there is only one queue in connection"
_ -> pure $ Left "sent QTEST: queue not in connection or not replacing another queue"
_ -> pure $ Left "QTEST sent not in duplex connection"
-- subQ write is now OUTSIDE connLock — blocking writeTBQueue is safe
case evt_ of
Right evt -> notify evt
Left err -> internalErr msgId err
```
All DB operations remain inside the lock. Only `notify`/`internalErr` (which write to subQ) move outside. `internalErr` calls `notifyDel` = `notify >> delMsg` — both `notify` (subQ write) and `delMsg` (`deleteSndMsgDelivery`, keyed on unique msgId) are safe outside the lock. The existing double-delete pattern (`delMsg` inside `internalErr` + `delMsgKeep` at line 2216) is preserved.
### Fix: Sites 2 & 3 — `ackQueueMessage::sendMsgNtf` (line 2386)
Change `ackQueueMessage` to return the MSGNTF event instead of writing it to subQ. Callers write to subQ after releasing connLock.
**Current code** (Agent.hs:2371-2386):
```haskell
ackQueueMessage :: AgentClient -> RcvQueue -> SMP.MsgId -> AM ()
ackQueueMessage c rq@RcvQueue {userId, connId, server} srvMsgId = do
atomically $ incSMPServerStat c userId server ackAttempts
tryAllErrors (sendAck c rq srvMsgId) >>= \case
Right _ -> sendMsgNtf ackMsgs
Left (SMP _ SMP.NO_MSG) -> sendMsgNtf ackNoMsgErrs
Left e -> ...
where
sendMsgNtf stat = do
atomically $ incSMPServerStat c userId server stat
whenM (liftIO $ hasGetLock c rq) $ do
atomically $ releaseGetLock c rq
brokerTs_ <- eitherToMaybe <$> tryAllErrors (withStore c $ \db -> getRcvMsgBrokerTs db connId srvMsgId)
atomically $ writeTBQueue (subQ c) ("", connId, AEvt SAEConn $ MSGNTF srvMsgId brokerTs_)
```
**New code** — return `Maybe ATransmission` instead of writing:
```haskell
ackQueueMessage :: AgentClient -> RcvQueue -> SMP.MsgId -> AM (Maybe ATransmission)
ackQueueMessage c rq@RcvQueue {userId, connId, server} srvMsgId = do
atomically $ incSMPServerStat c userId server ackAttempts
tryAllErrors (sendAck c rq srvMsgId) >>= \case
Right _ -> sendMsgNtf ackMsgs
Left (SMP _ SMP.NO_MSG) -> sendMsgNtf ackNoMsgErrs
Left e -> ... >> pure Nothing
where
sendMsgNtf stat = do
atomically $ incSMPServerStat c userId server stat
ifM (liftIO $ hasGetLock c rq)
(do atomically $ releaseGetLock c rq
brokerTs_ <- eitherToMaybe <$> tryAllErrors (withStore c $ \db -> getRcvMsgBrokerTs db connId srvMsgId)
pure $ Just ("", connId, AEvt SAEConn $ MSGNTF srvMsgId brokerTs_))
(pure Nothing)
```
**Caller 1: `ackMessage'`** (Agent.hs:2253-2267) — return event from `withConnLock`, write after:
```haskell
ackMessage' c connId msgId rcptInfo_ = do
t_ <- withConnLock c connId "ackMessage" $ do
SomeConn _ conn <- withStore c (`getConn` connId)
case conn of
DuplexConnection {} -> do
t_ <- ack
sendRcpt conn
del
pure t_
RcvConnection {} -> do
t_ <- ack
del
pure t_
SndConnection {} -> throwE $ CONN SIMPLEX "ackMessage"
ContactConnection {} -> throwE $ CMD PROHIBITED "ackMessage: ContactConnection"
NewConnection _ -> throwE $ CMD PROHIBITED "ackMessage: NewConnection"
-- subQ write is OUTSIDE connLock
case t_ of
Just t -> atomically $ writeTBQueue (subQ c) t
Nothing -> pure ()
```
**Caller 2: `ICAck` / `ICAckDel`** (Agent.hs:1823-1824) — inline `tryWithLock` as `tryCommand` + `withConnLock`, write subQ between the two scopes:
`tryWithLock name = tryCommand . withConnLock c connId name` — by inlining, the subQ write can be placed outside `withConnLock` but inside `tryCommand` (retaining retry/error handling).
```haskell
ICAck rId srvMsgId -> withServer $ \srv ->
tryCommand $ do
t_ <- withConnLock c connId "ICAck" $ ack srv rId srvMsgId
-- subQ write is OUTSIDE connLock — cannot deadlock with agentSubscriber
forM_ t_ $ atomically . writeTBQueue subQ
ICAckDel rId srvMsgId msgId -> withServer $ \srv ->
tryCommand $ do
t_ <- withConnLock c connId "ICAckDel" $ do
t_ <- ack srv rId srvMsgId
withStore' c (\db -> deleteMsg db connId msgId)
pure t_
-- subQ write is OUTSIDE connLock — cannot deadlock with agentSubscriber
forM_ t_ $ atomically . writeTBQueue subQ
```
Where `ack` now returns `AM (Maybe ATransmission)`:
```haskell
ack srv rId srvMsgId = do
rq <- withStore c $ \db -> getRcvQueue db connId srv rId
ackQueueMessage c rq srvMsgId
```
All subQ writes for MSGNTF are now outside connLock. FIFO ordering is preserved — no `nonBlockingWriteTBQueue`, no forked threads. The same thread that held the lock writes to subQ sequentially after releasing it.
### Race analysis
Window between connLock release and subQ write at Site 1:
| Thread | Can acquire connLock(X)? | Writes subQ? | Consequence |
|--------|-------------------------|-------------|-------------|
| agentSubscriber via sendMessagesB_ | Yes | **No** (encrypts only) | No race |
| processSMP for connId X | Yes | Yes (pending flush) | MSG before SWITCH — cosmetic |
| runCommandProcessing for connId X | Yes | Yes (pending flush) | Command response before SWITCH — cosmetic |
| New queue delivery worker | No (SENT outside lock) | Yes | SENT before SWITCH — cosmetic, **already exists in current code** |
All races are cosmetic UI ordering. None affect ratchet state, protocol correctness, or message delivery.
### Summary of changes
| File | Change | Lines affected |
|------|--------|---------------|
| Agent.hs | Restructure AM_QTEST_ to return event from `withConnLock`, write outside | ~2187-2214 |
| Agent.hs | Change `ackQueueMessage` return type to `AM (Maybe ATransmission)`, return event instead of writing | ~2371-2386 |
| Agent.hs | `ackMessage'`: return event from `withConnLock`, write outside | ~2253-2267 |
| Agent.hs | `ICAck`/`ICAckDel`: inline `tryCommand` + `withConnLock`, write subQ between scopes | ~1823-1824 |
| Agent.hs | `ack` helper: propagate new return type | ~1899-1901 |
No new data structures. No new modules. No changes to other write sites (1937, 3216 — already safe). ~25 lines changed total.
+620 -305
View File
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,65 @@
sequenceDiagram
participant A as Alice
participant AA as Alice's<br>agent
participant AS as Alice's<br>server
participant BS as Bob's<br>server
participant BA as Bob's<br>agent
participant B as Bob
note over AA, BA: status (receive/send): NONE/NONE
note over A, AA: 1. request connection<br>from agent
A ->> AA: createConnection
note over AA, AS: 2. create Alice's SMP queue
AA ->> AS: NEW: create SMP queue<br>allow sender to secure
AS ->> AA: IDS: SMP queue IDs
note over AA: status: NEW/NONE
AA ->> A: INV: invitation<br>to connect
note over A, B: 3. out-of-band invitation
A ->> B: OOB: invitation to connect
note over BA, B: 4. accept connection
B ->> BA: joinConnection:<br>via invitation info
note over BA: status: NONE/NEW
note over BA, AS: 5. secure Alice's SMP queue
BA ->> AS: SKEY: secure queue (this command needs to be proxied)
note over BA: status: NONE/SECURED
note over BA, BS: 6. create Bob's SMP queue
BA ->> BS: NEW: create SMP queue<br>allow sender to secure
BS ->> BA: IDS: SMP queue IDs
note over BA: status: NEW/SECURED
note over BA, AA: 7. confirm Alice's SMP queue
BA ->> AS: SEND: Bob's info without sender's key (SMP confirmation with reply queues)
note over BA: status: NEW/CONFIRMED
AS ->> AA: MSG: Bob's info without<br>sender server key
note over AA: status: CONFIRMED/NEW
AA ->> AS: ACK: confirm message
AA ->> A: CONF: connection request ID<br>and Bob's info
A -> AA: allowConnection: accept connection request,<br>send Alice's info
note over AA, BS: 8. secure Bob's SMP queue
AA ->> BS: SKEY: secure queue (this command needs to be proxied)
note over BA: status: CONFIRMED/SECURED
AA ->> BS: SEND: Alice's info without sender's server key (SMP confirmation without reply queues)
note over AA: status: CONFIRMED/CONFIRMED
note over AA, A: 9. notify Alice<br>about connection success<br>(no HELLO needed in v6)
AA ->> A: CON: connected
note over AA: status: ACTIVE/ACTIVE
note over BA, B: 10. notify Bob<br>about connection success
BS ->> BA: MSG: Alice's info without<br>sender's server key
note over BA: status: CONFIRMED/CONFIRMED
BA ->> B: INFO: Alice's info
BA ->> BS: ACK: confirm message
BA ->> B: CON: connected
note over BA: status: ACTIVE/ACTIVE
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 40 KiB

@@ -1,71 +0,0 @@
sequenceDiagram
participant A as Alice
participant AA as Alice's<br>agent
participant AS as Alice's<br>server
participant BS as Bob's<br>server
participant BA as Bob's<br>agent
participant B as Bob
note over AA, BA: status (receive/send): NONE/NONE
note over A, AA: 1. request connection<br>from agent
A ->> AA: NEW: create<br>duplex connection
note over AA, AS: 2. create Alice's SMP queue
AA ->> AS: NEW: create SMP queue
AS ->> AA: IDS: SMP queue IDs
note over AA: status: NEW/NONE
AA ->> A: INV: invitation<br>to connect
note over A, B: 3. out-of-band invitation
A ->> B: OOB: invitation to connect
note over BA, B: 4. accept connection
B ->> BA: JOIN:<br>via invitation info
note over BA: status: NONE/NEW
note over BA, BS: 5. create Bob's SMP queue
BA ->> BS: NEW: create SMP queue
BS ->> BA: IDS: SMP queue IDs
note over BA: status: NEW/NEW
note over BA, AA: 6. establish Alice's SMP queue
BA ->> AS: SEND: Bob's info and sender server key (SMP confirmation with reply queues)
note over BA: status: NEW/CONFIRMED
AS ->> AA: MSG: Bob's info and<br>sender server key
note over AA: status: CONFIRMED/NONE
AA ->> AS: ACK: confirm message
AA ->> A: CONF: connection request ID<br>and Bob's info
A ->> AA: LET: accept connection request,<br>send Alice's info
AA ->> AS: KEY: secure queue
note over AA: status: SECURED/NONE
AA ->> BS: SEND: Alice's info and sender's server key (SMP confirmation without reply queues)
note over AA: status: SECURED/CONFIRMED
BS ->> BA: MSG: Alice's info and<br>sender's server key
note over BA: status: CONFIRMED/CONFIRMED
BA ->> B: INFO: Alice's info
BA ->> BS: ACK: confirm message
BA ->> BS: KEY: secure queue
note over BA: status: SECURED/CONFIRMED
BA ->> AS: SEND: HELLO: only needs to be sent once in v2
note over BA: status: SECURED/ACTIVE
note over BA, B: 7a. notify Bob<br>about connection success
BA ->> B: CON: connected
AS ->> AA: MSG: HELLO: Alice's agent<br>knows Bob can send
note over AA: status: SECURED/ACTIVE
AA ->> AS: ACK: confirm message
note over A, AA: 7a. notify Alice<br>about connection success
AA ->> A: CON: connected
AA ->> BS: SEND: HELLO: only needs to be sent once in v2
note over AA: status: ACTIVE/ACTIVE
BS ->> BA: MSG: HELLO: Bob's agent<br>knows Alice can send
note over BA: status: ACTIVE/ACTIVE
BA ->> BS: ACK: confirm message
@@ -8,8 +8,8 @@ sequenceDiagram
note over AA, BA: status (receive/send): NONE/NONE
note over A, AA: 1. request connection from agent
A ->> AA: NEW: create<br>duplex connection
note over A, AA: 1. request connection<br>from agent
A ->> AA: createConnection
note over AA, AS: 2. create Alice's SMP queue
AA ->> AS: NEW: create SMP queue
@@ -17,63 +17,58 @@ sequenceDiagram
note over AA: status: NEW/NONE
AA ->> A: INV: invitation<br>to connect
note over AA: status: PENDING/NONE
note over A, B: 3. out-of-band invitation
A ->> B: OOB: invitation to connect
note over BA, B: 4. accept connection
B ->> BA: JOIN:<br>via invitation info
B ->> BA: joinConnection:<br>via invitation info
note over BA: status: NONE/NEW
note over BA, AA: 5. establish Alice's SMP queue
BA ->> AS: SEND: Bob's info and sender server key (SMP confirmation)
note over BA: status: NONE/CONFIRMED
activate BA
note over BA, BS: 5. create Bob's SMP queue
BA ->> BS: NEW: create SMP queue
BS ->> BA: IDS: SMP queue IDs
note over BA: status: NEW/NEW
note over BA, AA: 6. confirm Alice's SMP queue
BA ->> AS: SEND: Bob's info and sender server key (SMP confirmation with reply queues)
note over BA: status: NEW/CONFIRMED
AS ->> AA: MSG: Bob's info and<br>sender server key
note over AA: status: CONFIRMED/NONE
AA ->> AS: ACK: confirm message
AA ->> A: CONF: connection request ID<br>and Bob's info
A ->> AA: LET: accept connection request,<br>send Alice's info
A ->> AA: allowConnection: accept connection request,<br>send Alice's info
AA ->> AS: KEY: secure queue
note over AA: status: SECURED/NONE
BA ->> AS: SEND: HELLO: try sending until successful
deactivate BA
note over BA: status: NONE/ACTIVE
AS ->> AA: MSG: HELLO: Alice's agent<br>knows Bob can send
note over AA: status: ACTIVE/NONE
AA ->> AS: ACK: confirm message
AA ->> BS: SEND: Alice's info and sender's server key (SMP confirmation without reply queues)
note over AA: status: SECURED/CONFIRMED
note over BA, BS: 6. create Bob's SMP queue
BA ->> BS: NEW: create SMP queue
BS ->> BA: IDS: SMP queue IDs
note over BA: status: NEW/ACTIVE
note over AA, BA: 7. establish Bob's SMP queue
BA ->> AS: SEND: REPLY: invitation to the connect
note over BA: status: PENDING/ACTIVE
AS ->> AA: MSG: REPLY: invitation<br>to connect
note over AA: status: ACTIVE/NEW
AA ->> AS: ACK: confirm message
AA ->> BS: SEND: Alice's info and sender's server key
note over AA: status: ACTIVE/CONFIRMED
activate AA
note over BA, AA: 7. confirm Bob's SMP queue
BS ->> BA: MSG: Alice's info and<br>sender's server key
note over BA: status: CONFIRMED/ACTIVE
note over BA: status: CONFIRMED/CONFIRMED
BA ->> B: INFO: Alice's info
BA ->> BS: ACK: confirm message
BA ->> BS: KEY: secure queue
note over BA: status: SECURED/CONFIRMED
BA ->> AS: SEND: HELLO message
note over BA: status: SECURED/ACTIVE
AA ->> BS: SEND: HELLO: try sending until successful
deactivate AA
AS ->> AA: MSG: HELLO: Alice's agent<br>knows Bob can send
note over AA: status: SECURED/ACTIVE
AA ->> AS: ACK: confirm message
AA ->> BS: SEND: HELLO
note over A, AA: 8. notify Alice<br>about connection success
AA ->> A: CON: connected
note over AA: status: ACTIVE/ACTIVE
BS ->> BA: MSG: HELLO: Bob's agent<br>knows Alice can send
note over BA: status: ACTIVE/ACTIVE
BA ->> BS: ACK: confirm message
note over A, B: 8. notify users about connection success
AA ->> A: CON: connected
note over BA, B: 9. notify Bob<br>about connection success
BA ->> B: CON: connected
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 40 KiB

@@ -0,0 +1,17 @@
sequenceDiagram
participant A as Alice
participant R as Current server<br>that has A's<br>receive queue
participant R' as New server<br>that has the new A's<br>receive queue
participant S as Server<br>that has A's send queue<br>(B's receive queue)
participant B as Bob
A ->> R': NEW: create new queue<br>(allow SKEY)
A ->> S: SEND: QADD (R'): send address<br>of the new queue(s)
S ->> B: MSG: QADD (R')
B ->> R': SKEY: secure new queue
B ->> R': SEND: QTEST
R' ->> A: MSG: QTEST
A ->> R: DEL: delete the old queue
B ->> R': SEND: send messages to the new queue
R' ->> A: MSG: receive messages from the new queue
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 27 KiB

@@ -0,0 +1,21 @@
sequenceDiagram
participant A as Alice
participant R as Current server<br>that has A's<br>receive queue
participant R' as New server<br>that has the new A's<br>receive queue
participant S as Server<br>that has A's send queue<br>(B's receive queue)
participant B as Bob
A ->> R': NEW: create new queue
A ->> S: SEND: QADD (R'): send address<br>of the new queue(s)
S ->> B: MSG: QADD (R')
B ->> R: SEND: QKEY (R'): sender's key<br>for the new queue(s)
R ->> A: MSG: QKEY(R')
A ->> R': KEY: secure new queue
A ->> S: SEND: QUSE (R'): instruction to use new queue(s)
S ->> B: MSG: QUSE (R')
B ->> R': SEND: QTEST
R' ->> A: MSG: QTEST
A ->> R: DEL: delete the old queue
B ->> R': SEND: send messages to the new queue
R' ->> A: MSG: receive messages from the new queue
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 28 KiB

@@ -0,0 +1,30 @@
sequenceDiagram
participant M as mobile app
participant C as chat core
participant A as agent
participant P as push server
participant APN as APN
note over M, APN: get device token
M ->> APN: registerForRemoteNotifications()
APN ->> M: device token
note over M, P: register device token with push server
M ->> C: /_ntf register <token>
C ->> A: registerNtfToken(<token>)
A ->> P: TNEW
P ->> A: ID (tokenId)
A ->> C: registered
C ->> M: registered
note over M, APN: verify device token
P ->> APN: E2E encrypted code<br>in background<br>notification
APN ->> M: deliver background notification with e2ee verification token
M ->> C: /_ntf verify <e2ee code>
C ->> A: verifyNtfToken(<e2ee code>)
A ->> P: TVFY code
P ->> A: OK / ERR
A ->> C: verified
C ->> M: verified
note over M, APN: now token ID can be used
@@ -1,30 +1,26 @@
sequenceDiagram
participant M as mobile app
participant C as chat core
participant C as client app
participant A as agent
participant P as push server
participant APN as APN
participant P as SimpleX<br>Notification<br>Server
participant APN as Apple<br>Push Notifications<br>Server
note over M, APN: get device token
M ->> APN: registerForRemoteNotifications()
APN ->> M: device token
note over C, APN: get device token
C ->> APN: registerForRemoteNotifications()
APN ->> C: device token
note over M, P: register device token with push server
M ->> C: /_ntf register <token>
C ->> A: registerNtfToken(<token>)
note over C, P: register device token with push server
C ->> A: registerToken
A ->> P: TNEW
P ->> A: ID (tokenId)
A ->> C: registered
C ->> M: registered
note over M, APN: verify device token
note over C, APN: verify device token
P ->> APN: E2E encrypted code<br>in background<br>notification
APN ->> M: deliver background notification with e2ee verification token
M ->> C: /_ntf verify <e2ee code>
C ->> A: verifyNtfToken(<e2ee code>)
APN ->> C: deliver background notification with e2ee verification token
C ->> A: verifyToken<br>(<e2ee code>)
A ->> P: TVFY code
P ->> A: OK / ERR
A ->> C: verified
C ->> M: verified
note over M, APN: now token ID can be used
note over C, APN: now token ID can be used
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 28 KiB

@@ -0,0 +1,40 @@
sequenceDiagram
participant M as mobile app
participant C as chat core
participant A as agent
participant S as SMP server
participant N as NTF server
participant APN as APN
note over M, APN: register subscription
alt register existing
M -->> A: on /_ntf register, for subscribed queues
else create new connection
A -->> S: NEW / JOIN
note over A, S: ...<br>Connection handshake<br>...
S -->> A: CON
end
A ->> S: NKEY nKey
S ->> A: NID nId
A ->> N: SNEW tknId dhKey (smpServer, nId, nKey)
N ->> A: ID subId dhKey
N ->> S: NSUB nId
S ->> N: OK [/ NMSG]
note over M, APN: notify about message
S ->> N: NMSG
N ->> APN: APNSMutableContent<br>ntfQueue, nonce
APN ->> M: UNMutableNotificationContent
note over M, S: ...<br>Client awaken, message is received<br>...
S ->> M: message
note over M: mutate notification
note over M, APN: change APN token
APN ->> M: new device token
M -->> C: /_ntf_sub update tkn
C -->> A: updateNtfToken()
A -->> N: TUPD tknId newDeviceToken
note over M, N: ...<br>Verify token<br>...
@@ -1,17 +1,16 @@
sequenceDiagram
participant M as mobile app
participant C as chat core
participant C as client app
participant A as agent
participant S as SMP server
participant N as NTF server
participant APN as APN
note over M, APN: register subscription
note over C, APN: register subscription
alt register existing
M -->> A: on /_ntf register, for subscribed queues
C -->> A: registerToken
else create new connection
A -->> S: NEW / JOIN
A -->> S: create/joinConnection
note over A, S: ...<br>Connection handshake<br>...
S -->> A: CON
end
@@ -20,21 +19,20 @@ sequenceDiagram
A ->> N: SNEW tknId dhKey (smpServer, nId, nKey)
N ->> A: ID subId dhKey
N ->> S: NSUB nId
S ->> N: OK [/ NMSG]
S ->> N: OK / NMSG:<br>confirm subscription
note over M, APN: notify about message
note over C, APN: notify about message
S ->> N: NMSG
N ->> APN: APNSMutableContent<br>ntfQueue, nonce
APN ->> M: UNMutableNotificationContent
note over M, S: ...<br>Client awaken, message is received<br>...
S ->> M: message
note over M: mutate notification
APN ->> C: UNMutableNotificationContent
note over C, S: ...<br>Client awaken, message is received<br>...
S ->> C: message
note over C: show notification
note over M, APN: change APN token
note over C, APN: change APN token
APN ->> M: new device token
M -->> C: /_ntf_sub update tkn
C -->> A: updateNtfToken()
APN ->> C: new device token
C -->> A: updateToken()
A -->> N: TUPD tknId newDeviceToken
note over M, N: ...<br>Verify token<br>...
note over C, N: ...<br>Verify token<br>...
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 31 KiB

@@ -0,0 +1,23 @@
sequenceDiagram
participant B as Bob (sender)
participant S as server (queue RID)
participant A as Alice (recipient)
note over A: creating queue<br>("public" key RK<br>for msg retrieval)
A ->> S: 1. create queue ("NEW")
S ->> A: respond with queue RID and SID ("IDS")
note over A: out-of-band msg<br>(sender's queue SID<br>and "public" key EK<br>to encrypt msgs)
A -->> B: 2. send out-of-band message
note over B: secure queue<br>(with "public" key SK for<br>sending messages)
B ->> S: 3. confirm queue ("SKEY" command authorized with SK)
note over B: confirm queue<br>(public key<br>for e2e encryption<br>and any optional<br>encrypted info.)
B ->> S: 4. confirm queue ("SEND" command authorized with SK)
S ->> A: 5. deliver Bob's message (MSG)
note over A: decrypt message<br>("private" key EK)
A ->> S: acknowledge message (ACK)
note over S: 6. simplex<br>queue RID<br>is ready to use!
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 27 KiB

@@ -10,11 +10,13 @@ sequenceDiagram
note over A: out-of-band msg<br>(sender's queue SID<br>and "public" key EK<br>to encrypt msgs)
A -->> B: 2. send out-of-band message
note over B: confirm queue<br>("public" key SK for<br>sending messages<br>and any optional<br>info encrypted with<br>"public" key EK)
note over B: confirm queue<br>("public" key SK for<br>sending messages,<br>public key for<br>e2e encryption<br>and any optional<br>encrypted info)
B ->> S: 3. confirm queue ("SEND" command not signed)
S ->> A: 4. deliver Bob's message
S ->> A: 4. deliver Bob's message (MSG)
note over A: decrypt message<br>("private" key EK)
A ->> S: acknowledge message (ACK)
A ->> S: 5. secure queue ("KEY", RK-signed)
note over S: 6. simplex<br>queue RID<br>is ready to use!
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 26 KiB

@@ -0,0 +1,18 @@
sequenceDiagram
participant B as Bob (recipient)
participant S as XFTP server(s)
note over B: having received file description<br>from sender
loop for each chunk
B ->> S: 1a. download chunk ("FGET")
S ->> B: send chunk body ("FILE")
opt
B ->> S: 1b. acknowledge chunk reception ("FACK")
note over S: delete recipient ID
S ->> B: respond with ok ("OK")
end
end
note over B: 2. combine chunks into a file<br>3. decrypt file using key from file description<br>4. extract file name and unpad the file<br>5. validate file digest with the file description
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 23 KiB

@@ -0,0 +1,23 @@
sequenceDiagram
participant A as Alice (sender)
participant S as XFTP server(s)
participant B as recipient(s)
note over A: 1. prepare file:<br>encrypt,<br>split into chunks,<br>generate recipient<br>keys, etc.
loop for each chunk
A ->> S: 2a. register chunk ("FNEW")
S ->> A: respond with sender's and recipients' chunk IDs ("SIDS")
opt
A ->> S: 2b. request additional recipient IDs ("FADD")
S ->> A: respond with added recipients' chunk IDs ("RIDS")
end
A ->> S: 2c. upload chunk to chosen server ("FPUT")
S ->> A: respond with ok ("OK")
end
note over A: 3. prepare file description(s)
A -->> B: 4. send file description(s) out-of-band
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 25 KiB

+44
View File
@@ -0,0 +1,44 @@
sequenceDiagram
participant CI as Controller UI
participant CC as Controller Core
participant HC as Host Core
participant HI as Host UI
note over CI, HI: 1. Session invitation
CI->>CC: "Link a mobile"
CC-->>CI: Session invitation URI
note over CC: Listen for TCP connection
activate CC
HI->>HC: Session invitation URI
note over CI, HI: 2. Establishing TLS connection
HC-->>CC: TCP connect
note over CC, HC: TLS handshake
par
note over CC: validate client X509 credentials
CC->>CI: session code from tlsUnique
CI-->>CC: user confirmation
and
note over HC: validate server X509 credentials
HC->>HI: session code from tlsUnique
HI-->>HC: user confirmation
end
note over CI, HI: 3. Session verification and protocol negotiation
HC->>CC: host HELLO
note over CC: validate version, CA fingerprint
alt
CC-->>HC: controller ERROR
else
CC-->>HC: controller HELLO
note over CC, HC: update stored keys
end
deactivate CC
note over CI, HI: 4. Session operation
loop
CI->>CC: command
CC->>HC: XRCP command
HC-->>CC: XRCP response
CC-->>CI: response
end
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 31 KiB

+111 -181
View File
@@ -1,4 +1,4 @@
Revision 1, 2022-01-01
Revision 4, 2026-03-09
Evgeny Poberezkin
@@ -8,40 +8,47 @@ Evgeny Poberezkin
- [Introduction](#introduction)
- [What is SimpleX](#what-is-simplex)
- [Network model](#network-model)
- [Applications](#applications)
- [SimpleX objectives](#simplex-objectives)
- [In Comparison](#in-comparison)
- [Technical Details](#technical-details)
- [Trust in Servers](#trust-in-servers)
- [Client -> Server Communication](#client---server-communication)
- [Trust in Routers](#trust-in-routers)
- [Client -> Router Communication](#client---router-communication)
- [2-hop Onion Message Routing](#2-hop-onion-message-routing)
- [SimpleX Messaging Protocol](#simplex-messaging-protocol)
- [SimpleX Agents](#simplex-agents)
- [Encryption Primitives Used](#encryption-primitives-used)
- [Threat model](#threat-model)
- [Security](#security)
- [Acknowledgements](#acknowledgements)
## Introduction
#### What is SimpleX
SimpleX as a whole is a platform upon which applications can be built. [SimpleX Chat](https://github.com/simplex-chat/simplex-chat) is one such application that also serves as an example and reference application.
- [SimpleX Messaging Protocol](https://github.com/simplex-chat/simplexmq/blob/master/protocol/simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a server in-between. The messages are delivered via uni-directional queues created by recipients.
- [SimpleX Messaging Protocol](./simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a router in-between. The messages are delivered via uni-directional queues created by recipients.
- SMP protocol allows to send message via a SMP router playing proxy role using 2-hop onion routing (referred to as "private routing" in messaging clients) to protect transport information of the sender (IP address and session) from the router chosen (and possibly controlled) by the recipient.
- SMP runs over a transport protocol (shown below as TLS) that provides integrity, server authentication, confidentiality, and transport channel binding.
- A SimpleX Server is one of those servers.
- A SimpleX router is one of those routers.
- The SimpleX Network is the term used for the collective of SimpleX Servers that facilitate SMP.
- The SimpleX Network is the term used for the collective of SimpleX routers that facilitate SMP.
- SimpleX Client libraries speak SMP to SimpleX Servers and provide a low-level API not generally intended to be used by applications.
- SimpleX Client libraries speak SMP to SimpleX routers and provide a low-level API not generally intended to be used by applications.
- SimpleX Agents interface with SimpleX Clients to provide a more high-level API intended to be used by applications. Typically they are embedded as libraries, but are designed so they can also be abstracted into local services.
- SimpleX Agents interface with SimpleX Clients to provide a more high-level API intended to be used by applications. Typically they are embedded as libraries, but can also be abstracted into local services.
- SimpleX Agents communicate with other agents inside e2e encrypted envelopes provided by SMP protocol - the syntax and semantics of the messages exchanged by the agent are defined by [SMP agent protocol](./agent-protocol.md)
*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX Server.*
*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX router.*
```
User's Computer Internet Third-Party Server
User's Computer Internet Third-Party Router
------------------ | ---------------------- | -------------------------
| |
SimpleX Chat | |
@@ -51,11 +58,43 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
+----------------+ | |
| SimpleX Agent | | |
+----------------+ -------------- TLS ---------------- +----------------+
| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX Server |
| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX router |
+----------------+ ----------------------------------- +----------------+
| |
```
#### Network model
SimpleX is a general-purpose packet routing network built on top of the Internet. Network endpoints — end-user devices, automated services, AI-enabled applications, IoT devices — exchange data packets through SimpleX network nodes (SMP routers), which accept, buffer, and deliver packets. Each router operates independently and can be operated by any party on standard computing hardware.
SimpleX routers use resource-based addressing: each address identifies a resource on a router, similar to how the World Wide Web addresses resources via URLs. Internet routers, by comparison, use endpoint-based addressing, where IP addresses identify destination devices. Because of this design, SimpleX network participants do not need globally unique addresses to communicate.
SimpleX network has two resource-based addressing schemes:
- *Messaging queues* ([SMP](./simplex-messaging.md)). A queue is a unidirectional, ordered sequence of fixed-size data packets (16,384 bytes each). Each queue has a resource address on a specific router, gated by cryptographic credentials that separately authorize sending and receiving.
- *Data packets* ([XFTP](./xftp.md)). A data packet is an individually addressed block in one of the standard sizes. Each packet has a unique resource address on a specific router, gated by cryptographic credentials. Data packet addressing is more efficient for delivery of larger payloads than queues.
Packet delivery follows a two-router path. The sending endpoint submits a packet to a first router, which forwards it to a second router, where the receiving endpoint retrieves it. The sending endpoint's IP address is known only to the first router; the receiving endpoint's IP address is known only to the second router. See [2-hop Onion Message Routing](#2-hop-onion-message-routing) for details.
Routers buffer packets between submission and retrieval — from seconds to days, enabling asynchronous delivery when endpoints are online at different times. Packets are removed after delivery or after a configured expiration period.
#### Applications
Applications currently using SimpleX network:
- **SimpleX Chat** — a peer-to-peer messenger using SimpleX network as a transport layer, in the same way that communication applications use WebRTC, Tor, i2p, or Nym. All communication logic — contacts, conversations, groups, message formats, end-to-end encryption — runs on endpoint devices.
- **IoT devices** — using the SimpleX queue protocol directly for sensor data collection and device control.
- **AI-based services** — automated services built on the SimpleX Chat application core.
- **Secure monitoring and control systems** — applications for equipment monitoring and control, including robotics, using the network for command delivery and telemetry collection.
[SimpleGo](https://simplego.dev), developed by an independent organization, is a microcontroller-based device running a SimpleX Chat-compatible messenger directly on a microcontroller without a general-purpose operating system. Running over 20 days on a single battery charge, it demonstrates the energy efficiency of resource-based addressing: the device receives packets without continuous polling. A microcontroller-based router implementation that functions simultaneously as a WiFi router is also in development.
#### SimpleX objectives
1. Provide messaging infrastructure for distributed applications. This infrastructure needs to have the following qualities:
@@ -64,7 +103,7 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
- Privacy: protect against traffic correlation attacks to determine the contacts that the users communicate with.
- Reliability: the messages should be delivered even if some participating network servers or receiving clients fail, with at least once delivery guarantee.
- Reliability: the messages should be delivered even if some participating network routers or receiving clients fail, with "at least once" delivery guarantee.
- Integrity: the messages sent in one direction are ordered in a way that sender and recipient agree on; the recipient can detect when a message was removed or changed.
@@ -72,82 +111,104 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
- Low latency: the delay introduced by the network should not be higher than 100ms-1s in addition to the underlying TCP network latency.
2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious servers.
2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious routers.
3. Balance user experience with privacy requirements, prioritizing experience of mobile device users.
#### In Comparison
SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and servers without depending on any centralized component.
SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and routers without depending on any centralized component.
In comparison to more traditional messaging applications (e.g. WhatsApp, Signal, Telegram) the key differences of SimpleX network are:
- participants do not need to have globally unique addresses to communicate, instead they use redundant unidirectional (simplex) messaging queues, with a separate set of queues for each contact.
- connection requests are passed out-of-band, non-optionally protecting key exchange against man-in-the-middle attack.
- simple message queues provided by network servers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central servers, and content/communication channels.
- simple message queues provided by network routers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central routers, and content/communication channels.
- servers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence.
- routers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence.
- users can change routers with minimal disruption - even after an in-use router disappears, simply by changing the configuration on which routers the new queues are created.
- users can change servers with minimal disruption - even after an in-use server disappears, simply by changing the configuration on which servers the new queues are created.
## Technical Details
#### Trust in Servers
#### Trust in Routers
Clients communicate directly with servers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS.
Clients communicate directly with routers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS.
Users use multiple servers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen servers.
Users use multiple routers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen routers either directly, if this is a known/trusted router, or via another SMP router providing proxy functionality to protect IP address and session of the sender.
Although end-to-end encryption is always present, users place a degree of trust in servers. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX server is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) servers. While a user *may* re-use a connection to fetch from multiple queues, or connect to a server from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues.
Although end-to-end encryption is always present, users place a degree of trust in routers they connect to. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX router is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) routers. While a user *may* re-use a transport connection to fetch messages from multiple queues, or connect to a router from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues.
Users may trust a server because:
Users may trust a router because:
- They deploy and control the servers themselves from the available open-source code. This has the trade-offs of strong trust in the server but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that latter.
- They deploy and control the routers themselves from the available open-source code. This has the trade-offs of strong trust in the router but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that.
- They use servers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers.
- They use routers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers.
- Users trust their contacts and the servers they chose.
By default, routers do not retain access logs, and permanently delete messages and queues when requested. Messages persist in memory or in a database until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a router maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the user's communications graph and other meta-data.
By default, servers do not retain access logs, and permanently delete messages and queues when requested. Messages persist only in memory until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a server maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the users communications graph and other meta-data.
SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in routers. These include rotating the queues in use between users, noise traffic, supporting overlay networks such as Tor, and isolating traffic to different queues to different transport connections (and Tor circuits, if Tor is used).
SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in servers. These include rotating the queues in use between users, noise traffic, and supporting overlay networks such as Tor.
[0] While configurable by servers, a minimum value is enforced by the default software. SimpleX Agents provide redundant routing over queues to mitigate against message loss.
[0] While configurable by routers, a minimum value is enforced by the default software. SimpleX Agents can provide redundant routing over queues to mitigate against message loss.
#### Client -> Server Communication
#### Client -> Router Communication
Utilizing TLS grants the SimpleX Messaging Protocol (SMP) server authentication and metadata protection to a passive network observer. But SMP does not rely on the transport protocol for message confidentiality or client authentication. The SMP protocol itself provides end-to-end confidentiality, authentication, and integrity of messages between communicating parties.
Servers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients. The offline certificate signs an online certificate used in the transport protocol handshake. [0]
Routers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients, as part of the router address. The offline certificate signs an online certificate used in the transport protocol handshake. [0]
If the transport protocol's confidentiality is broken, incoming and outgoing messages to the server cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the server is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the server can one pass and correlate messages.
If the transport protocol's confidentiality is broken, incoming and outgoing messages to the router cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the router is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the router can one pass and correlate messages.
It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the server incorrectly cannot do anything with the SMP messages except drop them.
It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the router incorrectly cannot do anything with the SMP messages except drop them.
After the connection is established, the client sends blocks of a fixed size 16Kb, and the server replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX servers can be deployed as onion services and SimpleX clients can communicate with servers over Tor to further improve participants privacy.
After the connection is established, the client sends blocks of a fixed size 16KB, and the router replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX routers can be deployed as onion services and SimpleX clients can communicate with routers over Tor to further improve participants privacy.
By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a server simultaneously, the server acts as a (very) low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised.
By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a router simultaneously, the router acts as a low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised.
The protocol does not protect against attacks targeted at particular users with known identities - e.g., if the attacker wants to prove that two known users are communicating, they can achieve it. At the same time, it substantially complicates large-scale traffic correlation, making determining the real user identities much less effective.
The protocol does not protect against attacks targeted at particular users with known identities - e.g., if the attacker wants to prove that two known users are communicating, they can achieve it by observing their local traffic. At the same time, it substantially complicates large-scale traffic correlation, making determining the real user identities much less effective.
[0] Future versions of SMP may add support for revocation lists of certificates, presently this risk is mitigated by the SMP protocol itself.
#### 2-hop Onion Message Routing
As SimpleX Messaging Protocol routers providing messaging queues are chosen by the recipients, in case senders connect to these routers directly the router owners (who potentially can be the recipients themselves) can learn senders' IP addresses (if Tor is not used) and which other queues on the same router are accessed by the user in the same transport connection (even if Tor is used).
While the clients support isolating the messages sent to different queues into different transport connections (and Tor circuits), this is not practical, as it consumes additional traffic and system resources.
To mitigate this problem SimpleX Messaging Protocol routers support 2-hop onion message routing when the SMP router chosen by the sender forwards the messages to the routers chosen by the recipients, thus protecting both the senders IP addresses and sessions, even if connection isolation and Tor are not used.
The design of 2-hop onion message routing prevents these potential attacks:
- MITM by proxy (SMP router that forwards the messages).
- Identification by the proxy which and how many queues the sender sends messages to (as messages are additionally e2e encrypted between the sender and the destination SMP router).
- Correlation of messages sent to different queues via the same user session (as random correlation IDs and keys are used for each message).
See more details about 2-hop onion message routing design in [SimpleX Messaging Protocol](./simplex-messaging.md#proxying-sender-commands)
Also see [Security](./security.md)
#### SimpleX Messaging Protocol
SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a server (including IP, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public key for her receiving queue. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0]
SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a router (including IP address or host name, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public keys to agree e2e encryption. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0]
Because queues are uni-directional, Bob provides an identically-formatted introduction message to Alice over Alice's now-established receiving queue.
When setting up a queue, the server will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the server to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised.
When setting up a queue, the router will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the router to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised.
[0] Users can additionally create public 'contact queues' that are only used to receive connection requests.
[0] Users can additionally create public 'contact queues' that are only used to receive connection requests.
#### SimpleX Agents
SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with servers using SMP. Agent operations include:
SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with routers using SMP. Agent operations include:
- Managing sets of bi-directional, redundant queues for communication partners
@@ -157,153 +218,22 @@ SimpleX agents provide higher-level operations compared to SimpleX Clients, who
- Noise traffic
#### Encryption Primitives Used
- Ed448 to sign/verify commands to SMP servers (Ed25519 is also supported via client/server configuration).
- Curve25519 for DH exchange to agree:
- the shared secret between server and recipient (to encrypt message bodies - it avoids shared cipher-text in sender and recipient traffic)
- the shared secret between sender and recipient (to encrypt messages end-to-end in each queue - it avoids shared cipher-text in redundant queues).
- [NaCl crypto_box](https://nacl.cr.yp.to/box.html) encryption scheme (curve25519xsalsa20poly1305) for message body encryption between server and recipient and for E2E per-queue encryption.
- SHA256 to validate server offline certificates.
- [double ratchet](https://signal.org/docs/specifications/doubleratchet/) protocol for end-to-end message encryption between the agents:
- Curve448 keys to agree shared secrets required for double ratchet initialization (using [X3DH](https://signal.org/docs/specifications/x3dh/) key agreement with 2 ephemeral keys for each side),
- AES-GCM AEAD cipher,
- SHA512-based HKDF for key derivation.
## Security
## Threat Model
For encryption primitives, threat model, and detailed security analysis, see [Security](./security.md).
#### Global Assumptions
SimpleX provides these security properties:
- A user protects their local database and key material
- The user's application is authentic, and no local malware is running
- The cryptographic primitives in use are not broken
- A user's choice of servers is not directly tied to their identity or otherwise represents distinguishing information about the user.
- **End-to-end encryption** with forward secrecy via double ratchet protocol, with optional post-quantum protection.
#### A passive adversary able to monitor the traffic of one user
- **No shared identifiers** across connections — contacts cannot prove they communicate with the same user.
*can:*
- **Sender deniability** — neither routers nor recipients can cryptographically prove message origin.
- identify that and when a user is using SimpleX
- **Transport metadata protection** — fixed-size blocks, 2-hop onion routing, and connection isolation frustrate traffic correlation.
- block SimpleX traffic
- determine which servers the user communicates with
- observe how much traffic is being sent, and make guesses as to its purpose.
*cannot:*
- see who sends messages to the user and who the user sends the messages to
#### A passive adversary able to monitor a set of senders and recipients
*can:*
- identify who and when is using SimpleX
- learn which SimpleX Messaging Protocol servers are used as receive queues for which users
- learn when messages are sent and received
- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers
- observe how much traffic is being sent, and make guesses as to its purpose
*cannot, even in case of a compromised transport protocol:*
- perform traffic correlation attacks with any increase in efficiency over a non-compromised transport protocol
#### SimpleX Messaging Protocol server
*can:*
- learn when a queue recipient or sender is online
- know how many messages are sent via the queue (although some may be noise)
- perform queue correlation (matching multiple queues to a single user) via either a re-used transport connection, user's IP Address, or connection timing regularities
- learn a user's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used.
- drop all future messages inserted into a queue, detectable only over other, redundant queues
- lie about the state of a queue to the recipient and/or to the sender (e.g. suspended or deleted when it is not).
- spam a user with invalid messages
*cannot:*
- undetectably add, duplicate, or corrupt individual messages
- undetectably drop individual messages, so long as a subsequent message is delivered
- learn the contents of messages
- distinguish noise messages from regular messages except via timing regularities
- compromise the user's end-to-end encryption with an active attack
#### An attacker who obtained Alice's (decrypted) chat database
*can:*
- see the history of all messages exchanged by Alice with her communication partners
- see shared profiles of contacts and groups
- surreptitiously receive new messages sent to Alice via existing queues; until communication queues are rotated or the Double-Ratchet advances forward
- prevent Alice from receiving all new messages sent to her - either surreptitiously by emptying the queues regularly or overtly by deleting them
- send messages from the user to their contacts; recipients will detect it as soon as the user sends the next message, because the previous message hash wont match (and potentially wont be able to decrypt them in case they dont keep the previous ratchet keys).
*cannot:*
- impersonate a sender and send messages to the user whose database was stolen. Doing so requires also compromising the server (to place the message in the queue, that is possible until the Double-Ratchet advances forward) or the user's device at a subsequent time (to place the message in the database).
- undetectably communicate at the same time as Alice with her contacts. Doing so would result in the contact getting different messages with repeated IDs.
- undetectably monitor message queues in realtime without alerting the user they are doing so, as a second subscription request unsubscribes the first and notifies the second.
#### A users contact
*can:*
- spam the user with messages
- forever retain messages from the user
*cannot:*
- cryptographically prove to a third-party that a message came from a user (assuming the users device is not seized)
- prove that two contacts they have is the same user
- cannot collaborate with another of the user's contacts to confirm they are communicating with the same user
#### An attacker who observes Alice showing an introduction message to Bob
*can:*
- Impersonate Bob to Alice
*cannot:*
- Impersonate Alice to Bob
#### An attacker with Internet access
*can:*
- Denial of Service SimpleX messaging servers
- spam a user's public “contact queue” with connection requests
*cannot:*
- send messages to a user who they are not connected with
- enumerate queues on a SimpleX server
- **Out-of-band key exchange** — connection requests passed outside the network protect against MITM attacks.
## Acknowledgements
+299
View File
@@ -0,0 +1,299 @@
Version 1, 2024-06-22
# Post-quantum resistant augmented double ratchet algorithm (PQDR)
## Table of contents
- [Overview](#overview)
- [Comparison with the other approaches](#comparison-with-the-other-approaches)
- [PQXDH for post-quantum key agreement](#pqxdh-for-post-quantum-key-agreement) (Signal)
- [Hybrid Signal protocol for post-quantum encryption](#hybrid-signal-protocol-for-post-quantum-encryption) (Tutanota)
- [Augmented double ratchet algorithm](#augmented-double-ratchet-algorithm)
- [Double ratchet with encrypted headers augmented with double PQ KEM](#double-ratchet-with-encrypted-headers-augmented-with-double-pq-kem)
- [Initialization](#initialization)
- [Encrypting messages](#encrypting-messages)
- [Decrypting messages](#decrypting-messages)
- [Ratchet message wire format](#ratchet-message-wire-format)
- [Encrypted ratchet message](#encrypted-ratchet-message)
- [Encrypted message header](#encrypted-message-header)
- [Plaintext message header](#plaintext-message-header)
- [KEM state machine](#kem-state-machine)
- [Implementation considerations](#implementation-considerations)
- [Chosen KEM algorithm](#chosen-kem-algorithm)
- [Summary](#summary)
## Overview
It is a reasonable assumption that "record-now-decrypt-later" attacks are ongoing, so the users want to use cryptographic schemes for end-to-end encryption that are augmented with some post-quantum algorithm that is believed to be resistant to quantum computers.
SimpleX Chat uses [double-ratchet with header encryption](https://signal.org/docs/specifications/doubleratchet/#double-ratchet-with-header-encryption) to provide end-to-end encryption to messages and files. This document describes augmented algorithm with post-quantum key encapsulation mechanism (KEM) making it resistant to quantum computers.
Double-ratchet algorithm is a state of the art solution for end to end encryption offering a set of qualities that is not present in any other algorithm:
- perfect forward secrecy, i.e. compromise of session or long term keys does not lead to the ability to decrypt any of the past messages.
- deniability (also known as repudiation), i.e. the fact that the recipient of the message while having the proof of message authenticity, cannot prove to a third party that the sender actually sent this message.
- break-in recovery (also know as post-compromise security or future secrecy), i.e. the ability of the end-to-end encryption security to recover from the compromise of the long term keys. This is achieved by generating a new random key pair whenever a new DH key is received (DH ratchet step).
It is desirable to preserve all these qualities when augmenting the algorithm with a post-quantum algorithm, and having these qualities resistant to both conventional and quantum computers.
## Comparison with the other approaches
### PQXDH for post-quantum key agreement
[The solution](https://signal.org/docs/specifications/pqxdh/) recently [introduced by Signal](https://signal.org/blog/pqxdh/) augments the initial key agreement ([X3DH](https://signal.org/docs/specifications/x3dh/)) that is made prior to double ratchet algorithm. This is believed to provide protection from "record-now-decrypt-later" attack, but if the attacker at any point obtains long term keys from any of the devices, the break-in recovery will not be post-quantum resistant, and the attacker with quantum computer will be able to decrypt all the subsequent messages.
### Hybrid Signal protocol for post-quantum encryption
[The solution](https://eprint.iacr.org/2021/875.pdf) [proposed by Tutanota](https://tutanota.com/blog/posts/pqmail-update/) aims to preserve the break-in recovery property of double ratchet, but in doing so it:
- replaces rather than augments DH key agreement with post-quantum KEM mechanism, making it potentially vulnerable to conventional computers.
- adds signature to the DH ratchet step, to compensate for not keeping DH key agreement, but losing the deniability property for some of the messages.
## Augmented double ratchet algorithm
The double ratchet algorithm is augmented with post-quantum KEM mechanism, preserving all properties of the double ratchet algorithm.
It is possible, because although double ratchet uses DH (which is a non-interactive key exchanges), it uses it "interactively", when the new DH keys are generated by both parties in turns. Parties of double-ratchet encrypted communication can run two post-quantum key encapsulation mechanisms in parallel with both DH and KEM key agreements in each DH ratchet step, making break-in recovery of double ratchet algorithm post-quantum resistant, without losing deniability or resistance to conventional computers.
Specifically, [double ratchet with encrypted headers](https://signal.org/docs/specifications/doubleratchet/#double-ratchet-with-header-encryption) is augmented with some post-quantum key encapsulation mechanism (KEM) as described below. A possible algorithm for PQ KEM is [NTRU-prime](https://ntruprime.cr.yp.to), that is currently adopted in SSH and has available implementations. It is important though that the proposed scheme can be used with any PQ KEM algorithm.
The downside of the scheme is its substantial size overhead, as the encapsulation key and encapsulated shared secret are added to the header of each message. For the algorithm described below NTRU-prime adds ~2-4kb to each message (depending on the key size and the chosen variant). See [this table](https://ntruprime.cr.yp.to/security.html) for key and ciphertext sizes and the assessment of the security level for various key sizes.
It is possible to reduce size overhead by using only one KEM agreement and making only one of two ratchet steps providing post-quantum resistant break-in recovery.
## Double ratchet with encrypted headers augmented with double PQ KEM
Algorithm below assumes that in addition to shared secret from the initial key agreement, there will be an encapsulation key available from the party that published its keys (Bob).
### Initialization
The double ratchet initialization is defined in pseudo-code. This pseudo-code is identical to Signal algorithm specification except for that parts that add post-quantum key agreement.
```
// Alice obtained Bob's keys and initializes ratchet first
def RatchetInitAlicePQ2HE(state, SK, bob_dh_public_key, shared_hka, shared_nhkb, bob_pq_kem_encapsulation_key):
state.DHRs = GENERATE_DH()
state.DHRr = bob_dh_public_key
// below added for post-quantum KEM
state.PQRs = GENERATE_PQKEM()
state.PQRr = bob_pq_kem_encapsulation_key
state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext
// above added for KEM
// the next line augments DH key agreement with PQ shared secret
state.RK, state.CKs, state.NHKs = KDF_RK_HE(SK, DH(state.DHRs, state.DHRr) || state.PQRss)
state.CKr = None
state.Ns = 0
state.Nr = 0
state.PN = 0
state.MKSKIPPED = {}
state.HKs = shared_hka
state.HKr = None
state.NHKr = shared_nhkb
// Bob initializes ratchet second, having received Alice's connection request
def RatchetInitBobPQ2HE(state, SK, bob_dh_key_pair, shared_hka, shared_nhkb, bob_pq_kem_key_pair):
state.DHRs = bob_dh_key_pair
state.DHRr = None
// below added for KEM
state.PQRs = bob_pq_kem_key_pair
state.PQRr = None
state.PQRss = None
state.PQRct = None
// above added for KEM
state.RK = SK
state.CKs = None
state.CKr = None
state.Ns = 0
state.Nr = 0
state.PN = 0
state.MKSKIPPED = {}
state.HKs = None
state.NHKs = shared_nhkb
state.HKr = None
state.NHKr = shared_hka
```
`GENERATE_PQKEM` generates decapsulation/encapsulation key pair.
`PQKEM-ENC` is key encapsulation algorithm.
Other than commented lines, the above adds parameters `bob_pq_kem_encapsulation_key` and `bob_pq_kem_key_pair` to the ratchet initialization. Otherwise it is identical to the original double ratchet initialization.
### Encrypting messages
```
def RatchetEncryptPQ2HE(state, plaintext, AD):
state.CKs, mk = KDF_CK(state.CKs)
// encapsulation key from PQRs and encapsulated shared secret is added to header
header = HEADER_PQ2(
dh = state.DHRs.public,
kem = state.PQRs.public, // added for KEM #2
ct = state.PQRct // added for KEM #1
pn = state.PN,
n = state.Ns,
)
enc_header = HENCRYPT(state.HKs, header)
state.Ns += 1
return enc_header, ENCRYPT(mk, plaintext, CONCAT(AD, enc_header))
```
Other than adding encapsulation key and encapsulated shared secret into the header, the above is identical to the original double ratchet message encryption step.
### Decrypting messages
```
def RatchetDecryptPQ2HE(state, enc_header, ciphertext, AD):
plaintext = TrySkippedMessageKeysHE(state, enc_header, ciphertext, AD)
if plaintext != None:
return plaintext
header, dh_ratchet = DecryptHeader(state, enc_header) // DecryptHeader is the same as in double ratchet specification
if dh_ratchet:
SkipMessageKeysHE(state, header.pn) // SkipMessageKeysHE is the same as in double ratchet specification
DHRatchetPQ2HE(state, header)
SkipMessageKeysHE(state, header.n)
state.CKr, mk = KDF_CK(state.CKr)
state.Nr += 1
return DECRYPT(mk, ciphertext, CONCAT(AD, enc_header))
// DecryptHeader is the same as in double ratchet specification
def DecryptHeader(state, enc_header):
header = HDECRYPT(state.HKr, enc_header)
if header != None:
return header, False
header = HDECRYPT(state.NHKr, enc_header)
if header != None:
return header, True
raise Error()
def DHRatchetPQ2HE(state, header):
state.PN = state.Ns
state.Ns = 0
state.Nr = 0
state.HKs = state.NHKs
state.HKr = state.NHKr
state.DHRr = header.dh
// save new encapsulation key from header
state.PQRr = header.kem
// decapsulate shared secret from header - KEM #2
ss = PQKEM-DEC(state.PQRs.private, header.ct)
// use decapsulated shared secret with receiving ratchet
state.RK, state.CKr, state.NHKr = KDF_RK_HE(state.RK, DH(state.DHRs, state.DHRr) || ss)
state.DHRs = GENERATE_DH()
// below is added for KEM
state.PQRs = GENERATE_PQKEM() // generate new PQ key pair
state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext KEM #1
// above is added for KEM
// use new shared secret with sending ratchet
state.RK, state.CKs, state.NHKs = KDF_RK_HE(state.RK, DH(state.DHRs, state.DHRr) || state.PQRss)
```
`PQKEM-DEC` is key decapsulation algorithm.
`DHRatchetPQ2HE` augments both DH agreements with decapsulated shared secret from the received header and with the new shared secret, respectively. The new shared secret together with the new encapsulation key are saved in the state and will be added to the header in the next sent message.
Other than augmenting DH key agreements with the shared secrets from KEM, the above is identical to the original double ratchet DH ratchet step.
It is worth noting that while DH agreements work as ping-pong, when the new received DH key is used for both DH agreements (and only the sent DH key is updated for the second DH key agreement), PQ KEM agreements in the proposed scheme work as a "parallel ping-pong", with two balls in play all the time (two KEM agreements run in parallel).
## Ratchet message wire format
The pseudocode above describes the algorithm. This section specifies the actual binary encoding used in SimpleX implementation with Curve448 DH keys, sntrup761 KEM and AES-256-GCM AEAD.
The ratchet-encrypted message has three encoding layers, from outermost to innermost:
1. **Encrypted ratchet message** — the complete ratchet message envelope, referenced as an opaque encrypted body in [agent protocol](./agent-protocol.md).
2. **Encrypted message header** — the encrypted header within the ratchet message, used as associated data for message body encryption.
3. **Plaintext message header** — the DH and KEM ratchet keys and counters.
### Encrypted ratchet message
The outer envelope contains the encrypted header (used as associated data for body authentication), the body authentication tag, and the encrypted message body.
The message body is encrypted with AES-256-GCM using the message key derived from the sending chain key (`KDF_CK`). The associated data for body encryption is the concatenation of the ratchet associated data and the encoded encrypted header.
```abnf
encRatchetMessage = versionedLength encMessageHeader msgAuthTag encMsgBody
; encMessageHeader is used as associated data for body decryption: AD = rcAD || encMessageHeader
msgAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the message body
encMsgBody = *OCTET ; AES-256-GCM encrypted padded message body (remaining bytes)
```
### Encrypted message header
The encrypted header wraps the current ratchet e2e encryption version, an initialization vector, an authentication tag, and the encrypted padded header body.
The header body is encrypted with AES-256-GCM using the header key (`HKs`). The associated data for header encryption is the ratchet associated data. The header is padded before encryption to a fixed size to prevent leaking information about the KEM state.
```abnf
encMessageHeader = currentVersion headerIV headerAuthTag versionedLength encHeaderBody
currentVersion = 2*2 OCTET ; Word16, current ratchet e2e encryption version
headerIV = 16*16 OCTET ; AES-256 initialization vector for header encryption
headerAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the header
encHeaderBody = *OCTET ; AES-256-GCM encrypted padded header (see plaintext format below)
```
`versionedLength` uses a 2-byte length prefix (Word16) when the current e2e version supports PQ encryption, or a 1-byte length prefix otherwise. The parser distinguishes the two encodings by peeking at the first byte: values below 32 indicate a 2-byte prefix (as the header is always at least 69 bytes).
```abnf
versionedLength = largeLength / length ; 2-byte for PQ versions, 1-byte for pre-PQ versions
```
The padded header sizes before encryption are: 2310 bytes when PQ is supported, 88 bytes when PQ is not supported. Padding uses a 2-byte big-endian length prefix followed by the plaintext header and `#` fill bytes.
### Plaintext message header
```abnf
msgHeader = maxVersion dhPublicKey [kemParams] prevMsgCount msgCount
maxVersion = 2*2 OCTET ; Word16, max supported e2e encryption version
dhPublicKey = length x509encoded ; Curve448 public DH ratchet key
kemParams = noKEM / proposedKEM / acceptedKEM
; present only when current ratchet version >= pqRatchetE2EEncryptVersion
noKEM = %x30 ; "0" - no KEM parameters
proposedKEM = %x31 %s"P" kemEncapsulationKey ; KEM proposed, not yet accepted
acceptedKEM = %x31 %s"A" kemCiphertext kemEncapsulationKey ; KEM accepted
kemEncapsulationKey = largeLength 1158*1158 OCTET ; sntrup761 encapsulation key
kemCiphertext = largeLength 1039*1039 OCTET ; sntrup761 ciphertext
prevMsgCount = 4*4 OCTET ; Word32, number of messages in previous sending chain
msgCount = 4*4 OCTET ; Word32, message number in current sending chain
length = 1*1 OCTET
largeLength = 2*2 OCTET ; Word16
```
### KEM state machine
PQ encryption can be enabled or disabled during a connection's lifetime. The KEM parameters in the header reflect three states:
- **No KEM** (`noKEM`): PQ encryption is not active. The header contains only the DH key, as in the original double ratchet.
- **Proposed** (`proposedKEM`): One party generated a KEM key pair and includes the encapsulation key in the header, proposing PQ encryption. No ciphertext is included because the other party has not yet sent its encapsulation key.
- **Accepted** (`acceptedKEM`): The party received the other's encapsulation key, performed encapsulation (KEM #1), and includes both the ciphertext and its own new encapsulation key (for KEM #2). This is the steady state for active PQ encryption.
The transition from Proposed to Accepted happens when a party receives a message containing KEM parameters (either Proposed or Accepted) and responds with its own Accepted parameters. Once both parties are in Accepted state, the double PQ KEM augmentation described in the algorithm above operates in each DH ratchet step.
## Implementation considerations for SimpleX Messaging Protocol
As SimpleX Messaging Protocol pads messages to a fixed size, using 16kb transport blocks, the size increase introduced by this scheme can be compensated for by using ZSTD encryption of JSON bodies and image previews encoded as base64. While there may be some rare cases of random texts that would fail to compress, in all real scenarios it would not cause the message size reduction.
Sharing the initial keys in case of SimpleX Chat it is equivalent to sharing the invitation link. As encapsulation key is large, it may be inconvenient to share it in the link in some contexts, e.g. when QR codes are used.
It is possible to postpone sharing the encapsulation key until the first message from Alice (confirmation message in SMP protocol), the party sending connection request. The upside here is that the invitation link size would not increase. The downside is that the user profile shared in this confirmation will not be encrypted with PQ-resistant algorithm.
Another consideration is pairwise ratchets in groups. Key generation in sntrup761 is quite slow - on slow devices it can be as slow as 10-20 keys per second, so using this primitive in groups larger than 10-20 members would result in slow performance.
For backward compatibility the implementation must support adding PQ-resistant key agreement to the existing connections.
It is also beneficial to support removing PQ-resistant key agreement from the connections that have them, e.g. as the group size grows.
### Chosen KEM algorithm
The implementation uses Streamlined NTRU-Prime 761 (sntrup761) that was also used for OpenSSH for a long time.
It was chosen over ML-KEM (Kyber) standardized by NIST for several reasons:
- sntrup761 was used in OpenSSH for a long period of time.
- ML-KEM standardization process raised [concerns](https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/WFRDl8DqYQ4) [amongst](https://blog.cr.yp.to/20231003-countcorrectly.html) the experts.
- ML-KEM (if modified) is likely to have conflicts with the existing patents, unlike sntrup761.
It was chosen over non-interactive CTIDH due to its slower implementation, and lack of optimized code for aarch64 CPUs used in mobile devices.
## Summary
If chosen PQ KEM proves secure against quantum computer attacks, then the proposed augmented double ratchet will also be secure against quantum computer attack, including break-in recovery property, while keeping deniability and forward secrecy, because the [same proof](https://eprint.iacr.org/2016/1013.pdf) as for double ratchet algorithm would hold here, provided chosen KEM is secure.
+420
View File
@@ -0,0 +1,420 @@
Version 3, 2025-01-24
# Overview of push notifications for SimpleX Messaging Routers
This document describes Notification Router protocol version 3. Version history:
- v1: initial version
- v2: authenticated commands, command batching
- v3: detailed invalid token reason
## Table of contents
- [Introduction](#introduction)
- [Participating routers](#participating-routers)
- [Register device token to receive push notifications](#register-device-token-to-receive-push-notifications)
- [Subscribe to connection notifications](#subscribe-to-connection-notifications)
- [SimpleX Notification Router protocol](#simplex-notification-router-protocol)
- [Register new notification token](#register-new-notification-token)
- [Verify notification token](#verify-notification-token)
- [Check notification token status](#check-notification-token-status)
- [Replace notification token](#replace-notification-token)
- [Delete notification token](#delete-notification-token)
- [Subscribe to periodic notifications](#subscribe-to-periodic-notifications)
- [Create SMP message notification subscription](#create-smp-message-notification-subscription)
- [Check notification subscription status](#check-notification-subscription-status)
- [Delete notification subscription](#delete-notification-subscription)
- [Error responses](#error-responses)
- [Threat model](#threat-model)
## Introduction
SimpleX Messaging routers already operate as push routers and deliver the messages to subscribed clients as soon as they are sent to the routers.
The reason for push notifications is to support instant message notifications on iOS that does not allow background services.
## Participating routers
The diagram below shows which routers participate in message notification delivery.
While push provider (e.g., APN) can learn how many notifications are delivered to the user, it cannot access message content, even encrypted, or any message metadata - the notifications are e2e encrypted between SimpleX Notification Router and the user's device.
```
User's iOS device Internet Routers
--------------------- . ------------------------ . -----------------------------
. .
. . can be self-hosted now
+--------------+ . . +----------------+
| SimpleX Chat | -------------- TLS --------------- | SimpleX |
| client |------> SimpleX Messaging Protocol (SMP) ------> | Messaging |
+--------------+ ---------------------------------- | Router |
^ | . . +----------------+
| | . . . . . | . . .
| | . . | V |
| | . . |SMP| TLS
| | . . | | | SimpleX
| | . . . . . V . . . NTF Router
| | . . +----------------------------------+
| | . . | +---------------+ |
| | -------------- TLS --------------- | | SimpleX | can be |
| |-----------> Notification Router Protocol -----> | | Notifications | self-hosted |
| ---------------------------------- | | Subscriber | in the future |
| . . | +---------------+ |
| . . | | |
| . . | V |
| . . | +---------------+ |
| . . | | SimpleX | |
| . . | | Push | |
| . . | | Router | |
| . . | +---------------+ |
| . . +----------------------------------+
| . . . . . | . . .
| . . | V |
| . . |SMP| TLS
| . . | | |
| . . . . . V . . .
| -------------- TLS --------------- +-----------------+
|----------------- Notification delivery <-------| Apple PN server |
---------------------------------- +-----------------+
. .
```
## Register device token to receive push notifications
This diagram shows the process of registering a device to receive PUSH notifications via Apple Push Notification (APN) servers.
![Register device notification token](./diagrams/notifications/register-token.svg)
## Subscribe to connection notifications
This diagram shows the process of subscription to notifications, notification delivery and device token update.
![Subscribe to notifications](./diagrams/notifications/subscription.svg)
## SimpleX Notification Router protocol
To manage notification subscriptions to SMP routers, SimpleX Notification Router provides an RPC protocol with a similar design to SimpleX Messaging Protocol router.
This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the router certificate is not included in the handshake).
The client and router use ALPN extension with `ntf/1` protocol name to agree handshake version.
Protocol commands have this syntax:
```abnf
ntfRouterTransmission = authorization corrId entityId ntfRouterCmd
; same transmission structure as SMP, see simplex-messaging.md
ntfRouterCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd /
replaceTokenCmd / deleteTokenCmd / cronCmd /
newSubCmd / checkSubCmd / deleteSubCmd / pingCmd
```
### Register new notification token
This command should be used after the client app obtains a token from push notifications provider to register the token with the router.
Having received this command the router will deliver a test notification via the push provider to validate that the client has this token.
The command syntax:
```abnf
newTokenCmd = %s"TNEW" SP newToken
newToken = %s"T" deviceToken authPubKey clientDhPubKey
deviceToken = pushProvider tokenString
pushProvider = apnsDev / apnsProd / apnsTest / apnsNull
apnsDev = "AD" ; APNS token for development environment
apnsProd = "AP" ; APNS token for production environment
apnsTest = "AT" ; APNS token for test environment (mock server)
apnsNull = "AN" ; token that does not trigger any notification delivery - used for router testing
tokenString = shortString
authPubKey = length x509encoded ; Ed25519 key used to verify clients commands
clientDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client
shortString = length *OCTET
length = 1*1 OCTET
```
The router response syntax:
```abnf
tokenIdResp = %s"IDTKN" SP entityId routerDhPubKey
entityId = shortString
routerDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client
```
### Verify notification token
This command is used to verify the token after the device receives the test notification from the push provider.
The command syntax:
```abnf
verifyTokenCmd = %s"TVFY" SP regCode
regCode = shortString
```
The response to this command is `okResp` or `errorResp`
```abnf
okResp = %s"OK"
```
### Check notification token status
This command is used to check the token status:
```abnf
checkTokenCmd = %s"TCHK"
```
The response to this command:
```abnf
tokenStatusResp = %s"TKN" SP tokenStatus
tokenStatus = %s"NEW" / %s"REGISTERED" / tokenInvalid / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED"
tokenInvalid = %s"INVALID" ["," invalidReason] ; optional reason added in v3
invalidReason = %s"BAD" / %s"TOPIC" / %s"EXPIRED" / %s"UNREGISTERED"
```
### Replace notification token
This command should be used when push provider issues a new notification token.
It happens when:
- the app data is migrated to another device.
- the app is re-installed on the same device.
- can happen periodically, at push provider discretion.
This command allows to replace the token without re-registering and re-subscribing all notification subscriptions.
Using this command triggers the same verification flow as registering a new token.
The command syntax:
```abnf
replaceTokenCmd = %s"TRPL" SP deviceToken
```
The response to this command is `okResp` or `errorResp`.
### Delete notification token
The command syntax:
```abnf
deleteTokenCmd = %s"TDEL"
```
The response to this command is `okResp` or `errorResp`.
After this command all message notification subscriptions will be removed and no more notifications will be sent.
### Subscribe to periodic notifications
This command enables or disables periodic notifications sent to the client device irrespective of message notifications.
This is useful for two reasons:
- it provides better privacy from notification router, as while the router learns the device token, it doesn't learn anything else about user communications.
- it allows to receive messages when notifications were dropped by push provider, e.g. while the device was offline, or lost by notification router, e.g. while it was restarting.
The command syntax:
```abnf
cronCmd = %s"TCRN" SP interval
interval = 2*2 OCTET ; Word16, minutes
```
The interval for periodic notifications is set in minutes, with the minimum of 20 minutes. The client should pass `0` to disable periodic notifications.
### Create SMP message notification subscription
This command makes notification router subscribe to message notifications from SMP router and to deliver them to push provider:
```abnf
newSubCmd = %s"SNEW" SP newSub
newSub = %s"S" tokenId smpRouter notifierId notifierKey
tokenId = shortString ; returned in response to `TNEW` command
smpRouter = hosts port fingerprint
hosts = length 1*host
host = shortString
port = shortString
fingerprint = shortString
notifierId = shortString ; returned by SMP router in response to `NKEY` SMP command
notifierKey = length x509encoded ; private key used to authorize requests to subscribe to message notifications
```
The response syntax:
```abnf
subIdResp = %s"IDSUB" SP entityId
```
### Check notification subscription status
This command syntax:
```abnf
checkSubCmd = %s"SCHK"
```
The response:
```abnf
subStatusResp = %s"SUB" SP subStatus
subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP router disconnect/timeout while ntf router is retrying to connect
%s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another router subscribed to notifications
%s"AUTH" / %s"DELETED" / %s"SERVICE" / subErrStatus
subErrStatus = %s"ERR" SP *OCTET
```
### Delete notification subscription
The command syntax:
```abnf
deleteSubCmd = %s"SDEL"
```
The response to this command is `okResp` or `errorResp`.
After this command no more message notifications will be sent from this queue.
### Keep-alive command
To keep the transport connection alive the clients should use `PING` command:
```abnf
pingCmd = %s"PING"
pongResp = %s"PONG"
```
This command is sent unsigned and without entity ID.
### Error responses
All commands can return error response:
```abnf
errorResp = %s"ERR" SP errorType
```
Where `errorType` has the same syntax as in [SimpleX Messaging Protocol](./simplex-messaging.md#error-responses)
## Threat Model
This threat model compliments SimpleX Messaging Protocol [threat model](./security.md#threat-model)
#### A passive adversary able to monitor the traffic of one user
*can:*
- identify that and a user is using SimpleX push notifications.
*cannot:*
- determine which routers a user subscribed to the notifications from.
#### A passive adversary able to monitor a set of senders and recipients
*can:*
- perform more efficient traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers.
#### SimpleX Messaging Protocol router
*can:*
- learn which messages trigger push notifications.
- learn IP address of SimpleX notification routers used by the user.
- drop message notifications.
- spam a user with invalid notifications.
*cannot:*
- learn user device token for push notifications.
- learn which queues belong to the same users with any additional efficiency compared with not using push notifications.
#### SimpleX Notification Router subscribed to message notifications
*can:*
- learn a user device token.
- learn how many messaging queues and routers a user receives messages from.
- learn how many message notifications are delivered to the user from each queue.
- undetectably drop notifications.
- spam a user with background notifications.
*cannot:*
- learn queue addresses for receiving or sending messages.
- learn the contents or type of messages (not even encrypted).
- learn anything about messages sent without notification flag.
- spam a user with visible notifications (provided the client app can filter push notifications).
- add, duplicate, or corrupt individual messages that will be shown to the user.
#### SimpleX Notification Router subscribed ONLY to periodic notifications
*can:*
- learn a user device token.
- drop periodic notifications.
- spam a user with background notifications.
*cannot:*
- learn how many messaging queues and routers a user receives messages from.
- learn how many message notifications are delivered to the user from each queue.
- learn queue addresses for receiving or sending messages.
- learn the contents or type of messages (not even encrypted).
- learn anything about messages sent without notification flag.
- spam a user with visible notifications (provided the client app can filter push notifications).
- add, duplicate, or corrupt individual messages that will be shown to the user.
#### A users contact
*cannot:*
- determine if a user uses push notifications or not.
#### Push notification provider (e.g., APN)
*can:*
- learn that a user uses SimpleX app.
- learn how many notifications are delivered to user's device.
- drop notifications (in fact, APN coalesces notifications delivered while user's device is offline, delivering only the last one).
*cannot:*
- learn which SimpleX Messaging Protocol routers are used by a user (notifications are e2e encrypted).
- learn which or how many messaging queues a user receives notifications from.
- learn the contents or type of messages (not even encrypted, notifications only contain encrypted metadata).
#### An attacker with Internet access
*cannot:*
- register notification token not present on attacker's device.
- enumerate tokens or subscriptions on a SimpleX Notification Router.

Some files were not shown because too many files have changed in this diff Show More