Compare commits

...
77 Commits
Author SHA1 Message Date
Evgeny Poberezkin 27a37387be 7.0.1.0 2026-07-31 15:20:41 +01:00
sh d65d790a20 ntf server: shard push workers per notification token (#1840)
* ntf server: shard push workers per notification token

* ntf server: inline push worker shard calculation
2026-07-31 15:19:43 +01:00
shandEvgeny Poberezkin 7d0820dd44 smp server: do not create messaging queues in SMP proxy to prevent deadlock in processing (#1839)
* smp: fix proxy message queue memory leak

* smp server: do not create messaging queues in SMP proxy to prevent deadlock in processing

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-07-30 16:33:27 +01:00
Evgeny Poberezkin efaad8e734 7.0.0.6 2026-07-25 11:58:17 +01:00
Evgeny 1b4dcfe63e agent: refactor type for private keys in double ratchet (#1830) 2026-07-19 18:27:55 +01:00
Evgeny Poberezkin f7e8ed52bf 7.0.0.5 2026-07-18 06:35:31 +01:00
sh 399c5fe8c6 agent: pass optional SMP server to prepareConnectionLink (#1771) 2026-07-14 09:04:19 +01:00
sh 43e46dd8cc smp-server: fix service subscription memory leak (#1827)
Service subscription counters (totalServiceSubs, serviceSubsCount,
ntfServiceSubsCount) are TVar (Int64, IdsHash). modifyTVar' only forces
the pair to WHNF, so `n +/- n'` and `idsHash <> idsHash'` stay
unevaluated and accumulate an unbounded thunk chain under subscription
and delivery churn (the IdsHash chain also retains a bytestring per
update) - a space leak proportional to the number of updates.

Force both components in addServiceSubs/subtractServiceSubs. Verified
with the load bench: svc churn drops from +5.4 KiB/iter (linear) to flat.
2026-07-10 09:00:40 +01:00
Evgeny Poberezkin 551de8039f 7.0.0.4 2026-07-06 07:34:52 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> a45d764eaa agent: fix rare race conditions in async API (#1792)
* agent: fix rare race conditions in async API

* split async accept too

* fix, reduce diff

* composition

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-07-03 15:24:43 +01:00
Evgeny Poberezkin 836254a4c6 types: rename name types 2026-07-02 12:58:43 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 93925b257c types: instance for contact connection type (#1822)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-07-01 17:37:45 +01:00
Evgeny Poberezkin 6ef38a6ee7 7.0.0.3 2026-06-30 23:24:49 +01:00
shEvgenyEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
209f7826cb smp-server: support namespaces (#1784)
* smp-server: namespaces resolver scaffolding

* smp-server: Names resolver hardening + cleanup

* smp-server: fuse parallel dispatchers

* smp-server: JSON wire format for NameRecord + Names.hs restructure

* smp-server: redact RpcAuth in Show

* smp-server: JSON wire fixups + spec rewrite + small cleanups

* plan: prepend implementation-diverged banner

* move SimplexName into shared module

* smp-server: name + contract whitelist on RSLV

* smp-server: address audit findings (canonical JSON, INI guards, SSRF, TLD case, shutdown)

* smp-server: round 2 audit fixes (label case, response cap, ipv6 link-local)

* smp-server: round 3 audit fixes (SSRF coverage, drop noop closeManager, CSV order)

* smp-server: round 4 audit fixes (0X-hex host, expanded IPv6 forms, pingEndpoint timeout)

* smp-server: hardcode TldRegistries (drop registry_tld_* INI keys)

* smp-server: round 6 audit fixes (IPv6 SSRF, redirects, ASCII labels)

- Reject IPv6 aliases of 169.254.169.254 (IPv4-compatible / IPv4-mapped /
  6to4 / NAT64) via numeric range check on parsed IPv6.
- Disable HTTP redirects on the Eth RPC request.
- Restrict SimplexName labels to ASCII (Cyrillic/Greek/full-width otherwise
  hash to different on-chain records and diverge from UTS-46 registrars).
- pingEndpoint: only JsonRpcErr means "reachable"; transport/decode failures
  fail startup. boundedIniInt: readMaybe over partial read.
- Add 127.0.0.0/8 and 0.0.0.0 to isLoopback.
- Replace hand-rolled hex helpers with Data.ByteArray.Encoding; raise
  managerConnCount to match rpcMaxConcurrency; hex Show for NameOwner.
- Fuse parallel http/https when into unless+case; drop reverse/re-reverse
  in mkDomain TLDWeb; first AbiInvariantViolated; Nothing <$ decodeAddress;
  forM_ (eitherToMaybe ...); >>= chain in NameOwner FromJSON.
- Drop dead imports/exports/pragmas and two restating comments.
- Tests: factor unsafeOwner/unsafeLink, addr1/2/3, testNamesConfig; add
  non-ASCII label rejection coverage.

* namespace: bound parser input to 253 bytes (DoS defense)

The bare-name fallback and bareDomain parser would otherwise consume
arbitrarily many non-space bytes via takeWhile1 before any validation
or length check. A crafted multi-megabyte token would be decoded as
UTF-8 and re-parsed in full before being rejected.

Introduce `boundedNonSpace` (scan with 253-byte cap) at the two
takeWhile1 sites. Inputs longer than 253 bytes leave residue that
parseOnly's implicit endOfInput rejects, so the parser fails fast
without ever allocating the full input.

The bound is the DNS full-domain limit, chosen for being a familiar
ceiling generous enough to cover any realistic SimpleX name (longest
plausible @user.subdomain.simplex stays well under 100 bytes). No
per-label cap — SimpleX names don't go through DNS label resolution
and there's no semantic reason to constrain individual labels.

* namespace: switch to Python HTTP resolver + agent plumbing (#1796)

* namespace: relax resolver_endpoint validation (path prefix, http without auth)

validateUrl gains two operator-friendly relaxations and a regression test:

- Allow a path prefix (e.g. https://gw.example.com:443/snrc) for a resolver
  behind a reverse-proxy sub-path; /resolve/<name> and /health are appended
  (HttpResolver already strips one trailing slash, so root and sub-path
  behave identically). Query/fragment/userinfo stay rejected.

- Off-loopback, reject only http WITH resolver_auth (the Authorization header
  would travel in cleartext). http without auth is now allowed (no secret to
  leak; resolver data is public — also lets dev setups reach a host resolver
  via http://host.docker.internal). https is always allowed, with or without
  auth. Plain http has no response integrity; intended for trusted/local
  networks only.

Exports validateUrl and adds validateUrlSpec (11 cases) to SMPNamesTests.

* namespace: NameRecord links as arrays (multi-link, cap 5)

* namespace: distinct RSLV error responses

RSLV collapsed every non-hit (no resolver, malformed name, not found,
backing-store failure) to ERR AUTH, so a client iterating its configured
servers could not tell "this router has no resolver, try the next" from
"name not registered, stop", and a transient backend error read as an
authoritative miss.

Names capability is runtime config, orthogonal to the linear SMP version
(a future v21 router without [NAMES] must still advertise v21), so it is
signalled by a command-time error like allowSMPProxy, not by the version
range:

  no resolver configured -> ERR CMD PROHIBITED  (client skips, tries next)
  backing-store failure   -> ERR INTERNAL        (transient: retry/surface)
  not found / malformed   -> ERR AUTH            (authoritative "no such name")

Update the protocol spec error table and add agent tests for the
no-resolver (CMD PROHIBITED) and backend-failure (INTERNAL) paths.

* refactor(names): server role + one error type

Addresses epoberezkin's review (PR #1784). Name resolution becomes a
server role like proxy; the agent owns resolution + server selection;
one error type flows through the whole stack.

- ServerRoles gains `names`; UserServers gains `nameSrvs` (opt-in list);
  resolveSimplexName drops the explicit server arg and picks a
  names-capable server via getNextServer.
- RSLV carries SimplexNameDomain (was RslvRequest): no JSON on the wire,
  contract dropped, name validated at parse (invalid -> CMD SYNTAX).
- Version check moves from the encoder to Client.hs (no ERR to server).
- ErrorType.NAME {nameErr :: NameErrorType} (+ AgentErrorType.NAME),
  wire- and JSON-encoded; resolver errors surface with diagnostics.
  Success response renamed NAME -> RNAME to free the collision.
- NameOwner -> EthAddress (record selector); NameRecord derives FromJSON
  and gains field-ordered Encoding; per-field caps removed.
- Remove newEnvWithNames / runSMPServerBlockingWithNames test seams;
  stub resolver folded into ServerConfig.namesResolverCall_.

* test(server): update stats backup line count

NameResolverStatsData adds 6 lines to the server stats backup (the
"rslvStats:" header plus the reqs/succ/notFound/resolverErrs/disabled
fields), so testRestoreMessages' expected stats-backup line count is
95 -> 101.

* feat(names): public-namespace resolution via RSLV/RNAME

SNRC names resolver role: RSLV command -> HTTP resolver -> RNAME record.
Agent owns server selection (ServerRoles.names); NAME error family; async,
concurrency-bounded resolution; length-prefixed extensible wire; spec.

* remove comments

Co-authored-by: Evgeny <evgeny@poberezkin.com>

* simplify

* move tests name

* simplify: text addresses, Tail JSON, drop admitRslv

* fix

* remove spaghetti

* reduce diff

* async again, refactor

* different threads limit for name resolutions

* remove comment

* FromField instance for SimplexNameInfo

* remove comments

* unStrJSON

* add sameConnShortLink

* remove scheme prefix

* remove unused import

* remove connecttarget tests

* remove comment

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-06-30 22:54:55 +01:00
sh be58967a86 smp-server: fix subscriptions memory leak (#1820)
* docs: plan for service subscriptions memory leak

* fix: remove leaked service delivery subscriptions

The CSAEndServiceSub handler decremented subscription counters but did
not remove the per-queue delivery Sub from the service client's
subscriptions map. Over queue churn a long-lived service connection
accumulated orphaned Sub entries until disconnect, leaking memory.

Mirror CSAEndSub via endServiceQueueSub (reusing endSub) so the entry
is removed and its delivery thread cancelled.

Add a regression test with white-box access to the server Env via
runSMPServerBlocking_; verified failing without the fix.

* test: remove service subs leak regression test

Remove testServiceSubsRemovedOnQueueDelete and the test-only Env
exposure (runSMPServerBlocking_, withSmpServerConfigEnvOn,
serviceSubsMapSize), leaving only the server fix.

* refactor: simplify unsubPrev with applicative

Express the cancel-if-present logic as sequence_ (unsub_ <*> s_).
2026-06-30 15:38:45 +00:00
shandEvgeny Poberezkin c9ebf72e80 smp: fix proxy reconnection to relay after restart (#1806)
* tests: add SMP proxy relay reconnection tests

Reproduces the proxy failing to reconnect to a destination relay when the
sender disconnects mid-connection (empty session var left in smpClients).

* fix: bracket session var creation to drop it on interrupt

getSessVar inserts an empty session var that the connect path then fills with
putTMVar. If the connecting thread is killed by an async exception before that
fill (a proxy worker on client disconnect, an agent worker on cancel), the empty
var was left in the map forever and every later request for that server blocked
on it until timing out (permanent PCEResponseTimeout).

Wrap get-or-create with withGetSessVar (bracketOnError) at the call sites, so the
cleanup is established where the var is created and covers the whole connect: on
interrupt before fill the still-empty var is dropped and the next request
reconnects. This closes the window between getSessVar and the fill that a handler
installed inside the connect function cannot cover.

* test: cover session var leak on interrupted connect

UtilTests: tryAllErrors rethrows ThreadKilled/StackOverflow (the mechanism
that skips putTMVar). SMPProxyTests: agent client reconnection after a
cancelled connect, plus a control proving the stalling relay alone does not
cause the failure; refine the relay reconnection tests.

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-06-29 10:49:00 +00:00
sh 2dff11a808 resolver: cleanup (#1817)
* resolver: cleanup

* resolver: update .testing registry address
2026-06-23 16:30:44 +01:00
Evgeny Poberezkin 98391fd677 7.0.0.2 2026-06-21 13:28:46 +01:00
Evgeny Poberezkin d32a25c988 Merge branch 'stable' 2026-06-21 13:28:16 +01:00
EvgenyandPaul Bottinelli b2bdade380 fix: ignore pending XFTP files in storage accounting (#1814)
* fix: ignore pending XFTP files in storage accounting

* style

---------

Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-21 13:27:28 +01:00
Evgeny Poberezkin 92598c2ddb 6.5.5.0 2026-06-21 13:08:27 +01:00
Samy 84724bc03e crypto: validate BBS proof parameters (#1810) 2026-06-21 13:06:30 +01:00
Paul Bottinelli 91cb297e9e fix: disable web in cloud scripts without certs (#1804) 2026-06-21 12:52:08 +01:00
74a86043cc lib: parse bracketed IPv6 server addresses (#1807)
* Parse bracketed IPv6 server hosts

* lib: parse service-scheme and invitation hosts via TransportHost

* correct encoding

* encoding

---------

Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-06-21 12:18:34 +01:00
Evgeny 958de3bfca library: limit decompressed size (#1815) 2026-06-21 12:11:48 +01:00
EvgenyandPaul Bottinelli 45b21ec1db Reject duplicate STM short link updates (#1813)
Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-21 09:42:35 +01:00
EvgenyEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>Paul Bottinellish
aca1d9a462 crypto: sntrup length validation (#1811)
* Validate SNTRUP761 KEM input lengths

* crypto: BBS scheme for anonymous credentials with multiple presentations (#1794)

* crypto: BBS scheme for anonymous credentials with multiple presentations

* verify

* add files to sources

* more files

* more files, use cabal 3.0

* fix path

* extensions

* switch libbbs to fork

* return either from keygen

* use only secret key to sign

* improve FFI

* simplify

* update libbbs to support iOS

* add commoncrypto flag

* bump libbbs

* reject input of wrong length

* ci: get submodules

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* 6.5.4.0

* core: add getentropy shim for windows build (#1809)

* simplify

---------

Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-06-21 08:03:20 +01:00
brenziandsh 056314396d SNRC name resolver (#1795)
* add REST API to resolve SNRC

* fix unset fields

* support multi-TLD deployments

* update for mainnet tests

* haskell-friendly fieldnames

* add subname hint

* resolver: dockerize

* support multiple fallback links for splx contact and channels

* add test

* change url separator to semicolon

---------

Co-authored-by: sh <github.shum@liber.li>
2026-06-20 10:09:56 +01:00
Evgeny Poberezkin df6c53f830 7.0.0.1 2026-06-18 14:33:37 +01:00
Evgeny Poberezkin 220371cec1 Merge branch 'stable' 2026-06-18 14:29:31 +01:00
sh 44898bf7f6 core: add getentropy shim for windows build (#1809) 2026-06-18 14:28:55 +01:00
Evgeny Poberezkin 8e0b8de529 7.0.0.0 2026-06-17 17:14:36 +01:00
shandPaul Bottinelli db3e98f13a ntf-server: add push provider policy (#1808)
* Disable APNS test provider in production

* refactor(ntf): extract guardPushProvider for test-provider guard

* test(ntf): fix APNS test provider test compilation

* ntf server: use ifM for push provider guard (review)

---------

Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-17 09:14:38 +01:00
shandPaul Bottinelli 8a1b5608bf xftp-cli: add deprecation notice (#1799)
Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-16 10:07:13 +01:00
Evgeny Poberezkin e250a9ec9d Merge branch 'stable' 2026-06-16 06:50:14 +01:00
Evgeny Poberezkin 376d6a261a 6.5.4.0 2026-06-15 22:26:45 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 9f9b6c8e88 crypto: BBS scheme for anonymous credentials with multiple presentations (#1794)
* crypto: BBS scheme for anonymous credentials with multiple presentations

* verify

* add files to sources

* more files

* more files, use cabal 3.0

* fix path

* extensions

* switch libbbs to fork

* return either from keygen

* use only secret key to sign

* improve FFI

* simplify

* update libbbs to support iOS

* add commoncrypto flag

* bump libbbs

* reject input of wrong length

* ci: get submodules

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-06-15 09:44:11 +01:00
shandPaul Bottinelli 24e464926e scripts: fix check in simplex-servers-update (#1797)
Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-06 09:05:24 +01:00
shandPaul Bottinelli 7d3cfa56d3 xftp-web: remove debug logs (#1798)
Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-06 09:04:26 +01:00
sh 53bc0fe663 scripts: add docker-compose resolver setup (#1793) 2026-06-02 10:24:56 +01:00
Evgeny Poberezkin b981dcb70b 6.5.3.0 2026-06-01 13:17:47 +01:00
Evgeny Poberezkin 61ee188ee0 Merge branch 'stable' 2026-05-30 13:23:26 +01:00
Evgeny 39eb3c4a13 smp: fix handshake for rcv services between new client & old server (#1790) 2026-05-29 09:31:37 +01:00
sh ee2ff402fe agent: split SimplexNameDomain out of SimplexNameInfo (#1788)
* agent: split SimplexNameDomain out of SimplexNameInfo

The type now separates the user-supplied type prefix (#/@) from the
domain itself:

  data SimplexNameInfo = SimplexNameInfo
    { nameType :: SimplexNameType
    , nameDomain :: SimplexNameDomain
    }

  data SimplexNameDomain = SimplexNameDomain
    { nameTLD :: SimplexTLD
    , domain :: Text
    , subDomain :: [Text]
    }

The domain is independent of the contact-vs-public-group distinction —
the same dotted-labels structure applies to both. Future code that
needs to talk about a domain without committing to a name type (e.g.
server-side TLD-based registry lookup) can use SimplexNameDomain
directly.

fullDomainName now operates on SimplexNameDomain rather than the
full info wrapper. Parser, StrEncoding instance, and aeson derivations
updated accordingly. No external callers needed updating.

* agent: split StrEncoding instance for SimplexNameDomain

* agent: flatten TLD case + use unless guard

* agent: address review - strict domain parser, permissive channel
2026-05-29 09:11:08 +01:00
Evgeny Poberezkin 04960864c4 Merge branch 'stable' 2026-05-28 09:33:18 +01:00
Evgeny Poberezkin e9265a7f7c agent: allow all-digit names 2026-05-27 18:06:44 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 7682999505 agent: types for namespace support (#1786)
* agent: types for namespace support

* parser

* refactor

* more refactor

* simplify

* refactor again

* refactor

* refactor

* import

* use @ for contact addresses

* remove AConnectTarget

* update parser and types

* revert TLDWeb

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-05-27 13:05:13 +01:00
Evgenyspaced4ndyEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>sh
f0b7a4be73 messaging services (#1667)
* smp server: messaging services (#1565)

* smp server: refactor message delivery to always respond SOK to subscriptions

* refactor ntf subscribe

* cancel subscription thread and reduce service subscription count when queue is deleted

* subscribe rcv service, deliver sent messages to subscribed service

* subscribe rcv service to messages (TODO delivery on subscription)

* WIP

* efficient initial delivery of messages to subscribed service

* test: delivery to client with service certificate

* test: upgrade/downgrade to/from service subscriptions

* remove service association from agent API, add per-user flag to use the service

* agent client (WIP)

* service certificates in the client

* rfc about drift detection, and SALL to mark end of message delivery

* fix test

* fix test

* add function for postgresql message storage

* update migration

* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1668)

* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)

* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers

* smp server: xor hash with triggers

* fix sql and using pgcrypto extension in tests

* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands

* agent migrations with functions/triggers

* remove agent triggers

* try tracking service subs in the agent (WIP, does not compile)

* Revert "try tracking service subs in the agent (WIP, does not compile)"

This reverts commit 59e908100d.

* comment

* agent database triggers

* service subscriptions in the client

* test / fix client services

* update schema

* fix postgres migration

* update schema

* move schema test to the end

* use static function with SQLite to avoid dynamic wrapper

* agent: fail when per-connection transport isolation is used with services (#1670)

* agent: service subscription events (#1671)

* agent: use server keyhash when loading service record

* agent: process queue/service associations with delayed subscription results

* agent: service subscription events

* agent: finalize initial service subscriptions, remove associations on service ID changes (#1672)

* agent: remove service/queue associations when service ID changes

* agent: check that service ID in NEW response matches session ID in transport session

* agent subscription WIP

* test

* comment

* enable tests

* update queries

* agent: option to add SQLite aggregates to DB connection  (#1673)

* agent: add build_relations_vector function to sqlite

* update aggregate

* use static aggregate

* remove relations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add test, treat BAD_SERVICE as temp error, only remove queue associations on service errors

* add packZipWith for backward compatibility with GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* servers: service stats and logging, allow services without option (removed), report errors during service message delivery, remove threads when service subscription ended (#1676)

* smp server: always allow services without option

* smp server: maintain IDs hash in session subscription states

* smp server: service message delivery error handling

* ntf server: log subscription count and hash differences

* smp server: remove delivery threads when service subscription ended/client disconnected

* agent: remove service queue association when service ID changed, process ENDS event, test migrating to/from service (#1677)

* agent: remove service queue association when service ID changed

* agent: process ENDS event

* agent: send service subscription error event

* agent: test migrating to/from service subscriptions, fixes

* agent: always remove service when disabled, fix service subscriptions

* ntf server: use different client certs for each SMP server, remove support for store log (#1681)

* ntf server: remove support for store log

* ntf server: use different client certificates for each SMP server

* smp protocol: fix encoding for SOKS/ENDS responses (#1683)

* agent: create user with option to enable client service (#1684)

* agent: create user with option to enable client service

* handle HTTP2 errors

* do not catch async exceptions

* agent: minor fixes

* docs: update protocol (#1705)

* docs: agent threat model

* update protocol docs

* update RFCs (#1730)

* update RFCs

* update

* update overview

* update terminology

* original language in threat model

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* docs: fix minor issues in protocols

* docs: add e2e encrypted message wire encoding to PQDR spec

* docs: add missing encodings and other protocol corrections

* docs: move implemented rfcs

* smp: service fixes (#1737)

* smp: deliver service subscription to correct client

* tests: more resilient to concurrency

* optimize PostgreSQL query

* fix service re-association after server "downgrade"

* correctly handle service removed from server (and ID changed)

* remove unused

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* prometheus: fix metrics names (#1747)

* test: rcv service re-association on restart (#1746)

* agent: correct log message

* docs: update whitepaper

* smp: fix messaging client service issues (#1751)

* services: fix minor issues

* fix accounting for subscribed service queues, add prometheus stats

* fix uncorrelated subquery

* fix potential race condition when inserting service defensively, as it is also prevented by how client is created

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* agent: refactor cleanup if no pending subs (#1757)

* smp server: batch processing of subscription messages (#1753)

* smp server: batch processing of subscription messages

* refactor

* empty line

* fix

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* smp: batch queue association updates on subscriptions (#1760)

* smp: batch queue association updates on subscriptions

* refactor to fused batching

* simpler

* batch assoc functions

* clean up

* fix

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* agent: use primary key index in setRcvServiceAssocs (#1783)

* agent: use primary key index in setRcvServiceAssocs

Previous WHERE rcv_id = ? did not match the (host, port, rcv_id)
primary key prefix and fell back to a table scan via
idx_rcv_queues_client_notice_id. With ~390k rows per queue, each
update in a 1350-row batch scanned the whole table, yielding ~290s
per batch and a multi-hour rcv-services migration.

* agent: pass SMPServer explicitly to setRcvServiceAssocs

Avoid extracting host/port from the first queue inside setRcvServiceAssocs.
The caller already has SMPServer in scope (from tSess) and the call chain
is short, so threading it through is simpler than inspecting the list.
Removes the empty-list guard from setRcvServiceAssocs (it remains in
processRcvServiceAssocs).

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-05-21 14:14:03 +01:00
Evgeny Poberezkin f03cec7a58 6.5.2.0 2026-05-21 09:10:25 +01:00
sh b6f551000f ntf server: concurrent APNS push via sendRequestDirect (#1780)
The per-(srvHost, provider) worker shards added in #1779 still funnel
all APNS sends through one HTTP2Client's reqQ, where a single process
thread calls sendRequest serially - one in-flight HTTP/2 stream at a
time, capping APNS throughput at 1/RTT.

sendRequestDirect bypasses the queue and invokes sendReq directly from
the calling worker, so concurrent workers open parallel HTTP/2 streams
on the shared APNS connection and the multiplexing happens on the wire.
2026-05-18 14:35:47 +01:00
shEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>Evgeny Poberezkin
012c8cc104 ntf server: concurrent notification delivery (#1779)
* ntf-server: carry retry reason in PPRetryLater, log retries

Change PPRetryLater from nullary to PPRetryLater Text so the cause
(503 / 410-reason) propagates to the retry call site. Log a warning
at every retry attempt with provider, token id and reason.

* ntf-server: parallel push delivery via forkIO + per-srvHost lock

Fork delivery per notification, taking an MVar keyed by srvHost_ so
notifications from the same SMP server serialize while different
servers proceed concurrently. Switch APNS to sendRequestDirect so
concurrent deliveries share one HTTP/2 connection via stream
multiplexing rather than serializing through the client reqQ.

* ntf-server: single-flight push client creation via SessionVar

Match the take/create/wait pattern in Agent/Client.hs
(newProtocolClient / waitForProtocolClient). pushClients now wraps
clients in SessionVar (Either SomeException PushProviderClient) so
concurrent first-time access and concurrent retries collapse to a
single mkClient call; waiters observe the winner's result via
readTMVar (or its error). retryDeliver evicts the failing client by
SessionVar identity before re-fetching.

* use multiple queues and workers, remove semaphores and threads per notification

* fix

* retry connecting client

* fix

* move config

* fix

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-05-18 09:26:50 +01:00
sh fd298ae328 xftp: web page tweaks (#1772) 2026-05-06 09:24:18 +01:00
Evgeny Poberezkin 1f173abf6d 6.5.1.0 2026-05-01 17:57:19 +01:00
Evgeny 21f4597dad xftp: backwards compatible file header decoding (#1768) 2026-05-01 13:32:19 +01:00
Evgeny Poberezkin ba6af65c54 6.5.0.17 2026-04-29 20:31:42 +01:00
Evgeny Poberezkin 858fac7f4f 6.5.0.16 2026-04-21 20:26:51 +01:00
Evgeny Poberezkin 90432a44b4 tests: fix test compilation 2026-04-20 13:42:25 +01:00
sh 1e1f897c79 core: use = as INI key-value separator (#1767)
* core: use = as INI key-value separator

* core: update docker entrypoints for = INI separator

* core: update INI separator in README and test scripts
2026-04-20 09:22:14 +01:00
sh 0dc2940eff ci: add xftp-server postgres binaries (#1766) 2026-04-17 15:57:30 +01:00
sh 8833e5c1b5 xftp-server: support postgresql backend (#1755)
* xftp: add PostgreSQL backend design spec

* update doc

* adjust styling

* add implementation plan

* refactor: move usedStorage from FileStore to XFTPEnv

* refactor: add getUsedStorage, getFileCount, expiredFiles store functions

* refactor: change file store operations from STM to IO

* refactor: extract FileStoreClass typeclass, move STM impl to Store.STM

* refactor: make XFTPEnv and server polymorphic over FileStoreClass

* feat: add PostgreSQL store skeleton with schema migration

* feat: implement PostgresFileStore operations

* feat: add PostgreSQL INI config, store dispatch, startup validation

* feat: add database import/export CLI commands

* test: add PostgreSQL backend tests

* fix: map ForeignKeyViolation to AUTH in addRecipient

When a file is concurrently deleted while addRecipient runs, the FK
constraint on recipients.sender_id raises ForeignKeyViolation. Previously
this propagated as INTERNAL; now it returns AUTH (file not found).

* fix: only decrement usedStorage for uploaded files on expiration

expireServerFiles unconditionally subtracted file_size from usedStorage
for every expired file, including files that were never uploaded (no
file_path). Since reserve only increments usedStorage during upload,
expiring never-uploaded files caused usedStorage to drift negative.

* fix: handle setFilePath error in receiveServerFile

setFilePath result was discarded with void. If it failed (file deleted
concurrently, or double-upload where file_path IS NULL guard rejected
the second write), the server still reported FROk, incremented stats,
and left usedStorage permanently inflated. Now the error is checked:
on failure, reserved storage is released and AUTH is returned.

* fix: escape double quotes in COPY CSV status field

The status field (e.g. "blocked,reason=spam,notice={...}") is quoted in
CSV for COPY protocol, but embedded double quotes from BlockingInfo
notice (JSON) were not escaped. This could break CSV parsing during
import. Now double quotes are escaped as "" per CSV spec.

* fix: reject upload to blocked file in Postgres setFilePath

In Postgres mode, getFile returns a snapshot TVar for fileStatus. If a
file is blocked between getFile and setFilePath, the stale status check
passes but the upload should be rejected. Added status = 'active' to
the UPDATE WHERE clause so blocked files cannot receive uploads.

* fix: add CHECK constraint on file_size > 0

Prevents negative or zero file_size values at the database level.
Without this, corrupted data from import or direct DB access could
cause incorrect storage accounting (getUsedStorage sums file_size,
and expiredFiles casts to Word32 which wraps negative values).

* fix: check for existing data before database import

importFileStore now checks if the target database already contains
files and aborts with an error. Previously, importing into a non-empty
database would fail mid-COPY on duplicate primary keys, leaving the
database in a partially imported state.

* fix: clean up disk file when setFilePath fails in receiveServerFile

When setFilePath fails (file deleted or blocked concurrently, or
duplicate upload), the uploaded file was left orphaned on disk with
no DB record pointing to it. Now the file is removed on failure,
matching the cleanup in the receiveChunk error path.

* fix: check storeAction result in deleteOrBlockServerFile_

The store action result (deleteFile/blockFile) was discarded with void.
If the DB row was already deleted by a concurrent operation, the
function still decremented usedStorage, causing drift. Now the error
propagates via ExceptT, skipping the usedStorage adjustment.

* fix: check deleteFile result in expireServerFiles

deleteFile result was discarded with void. If a concurrent delete
already removed the file, deleteFile returned AUTH but usedStorage
was still decremented — causing double-decrement drift. Now the
usedStorage adjustment and filesExpired stat only run on success.

* refactor: merge STM store into Store.hs, parameterize server tests

- Move STMFileStore and its FileStoreClass instance from Store/STM.hs
  back into Store.hs — the separate file was unnecessary indirection
  for the always-present default implementation.

- Parameterize xftpFileTests over store backend using HSpec SpecWith
  pattern (following SMP's serverTests approach). The same 11 tests
  now run against both memory and PostgreSQL backends via a bracket
  parameter, eliminating all *Pg test duplicates.

- Extract shared run* functions (runTestFileChunkDeliveryAddRecipients,
  runTestWrongChunkSize, runTestFileChunkExpiration, runTestFileStorageQuota)
  from inlined test bodies.

* refactor: clean up per good-code review

- Remove internal helpers from Postgres.hs export list (withDB, withDB',
  handleDuplicate, assertUpdated, withLog are not imported externally)
- Replace local isNothing_ with Data.Maybe.isNothing in Env.hs
- Consolidate duplicate/unused imports in XFTPStoreTests.hs
- Add file_path IS NULL and status guards to STM setFilePath, matching
  the Postgres implementation semantics

* test: parameterize XFTP server, agent and CLI tests over store backend

- xftpTest/xftpTest2/xftpTest4/xftpTestN now take XFTPTestBracket as
  first argument, enabling the same test to run against both memory
  and PostgreSQL backends.

- xftpFileTests (server tests), xftpAgentFileTests (agent tests), and
  xftpCLIFileTests (CLI tests) are SpecWith-parameterized suites that
  receive the bracket from HSpec's before combinator.

- Test.hs runs each parameterized suite twice: once with
  xftpMemoryBracket, once with xftpPostgresBracket (CPP-guarded).

- STM-specific tests (store log restore/replay) stay in memory-only
  xftpAgentTests. SNI/CORS tests stay in memory-only xftpServerTests.

* refactor: remove dead test wrappers after parameterization

Remove old non-parameterized test wrapper functions that were
superseded by the store-backend-parameterized test suites.
All test bodies (run* and _ functions) are preserved and called
from the parameterized specs. Clean up unused imports.

* feat: add manual tests and guide

* refactor: merge file_size CHECK into initial migration

* refactor: extract rowToFileRec shared by getFile sender/recipient paths

* refactor: parameterize XFTPServerConfig over store type

Embed XFTPStoreConfig s as serverStoreCfg field, matching SMP's
ServerConfig. runXFTPServer and newXFTPServerEnv now take a single
XFTPServerConfig s. Restore verifyCmd local helper structure.

* refactor: minimize diff in tests

Restore xftpServerTests and xftpAgentTests bodies to match master
byte-for-byte (only type signatures change for XFTPTestBracket
parameterization); inline the runTestXXX helpers that were split
on this branch.

* refactor: restore getFile position to match master

* refactor: rename withSTMFile back to withFile

* refactor: close store log inside closeFileStore for STM backend

Move STM store log close responsibility into closeFileStore to
match PostgresFileStore, removing the asymmetry where only PG's
close was self-contained.

STMFileStore holds the log in a TVar populated by newXFTPServerEnv
after readWriteFileStore; stopServer no longer needs the explicit
withFileLog closeStoreLog call. Writes still go through XFTPEnv.storeLog
via withFileLog (unchanged).

* refactor: rename XFTPTestBracket to XFTPTestServer

* fix: move file_size check from PG schema to store log import

* refactor: use SQL-standard type names in XFTP schema

* perf: batch expired file deletions with deleteFiles

* refactor: stream export instead of loading recipients into memory

* refactor: parameterize XFTP store with FSType singleton dispatch

* refactor: minimize diff per review feedback

* refactor: use types over strings, deduplicate parser

* refactor: always parse database store type, fail at startup

* fix compilation without postgresql

* refactor: always parse database store type, fail at startup
2026-04-16 09:06:04 +01:00
Evgeny 95b17ada27 lib: fix incorrect encoding of Signature (incompatible with decoding, but never used together) - breaks backward compatibility for remote control connections (#1765)
* lib: fix incorrect StrEncoding of Signature (it was not compatible with decoding, but was never used)

* align encoding with used in links (breaks backward compatibility)
2026-04-15 15:11:06 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 43cdf55f3b lib: add JSON instance to Signature type (#1764)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-04-14 19:47:20 +01:00
Evgeny Poberezkin bc5ea42bec 6.5.0.15 2026-04-11 19:39:50 +01:00
Evgeny Poberezkin 0933cbcb9c agent: add compression api 2026-04-11 18:14:48 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> f2dafd983b agent: export decompressedSize (#1763)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-04-11 17:28:00 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 34c0909c1a agent: drop message after N reception attempts (#1762)
* agent: drop message after N reception attempts

* test

* increase count for message expiration

* fix migration

* update schema

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-04-11 16:24:30 +01:00
Evgeny Poberezkin 97802a30fc 6.5.0.14 2026-04-04 17:28:23 +01:00
Evgeny b82cf7d001 xftp: remove page (#1761) 2026-04-03 10:47:52 +01:00
spaced4ndy 9bc0c70fa0 agent: getConnLinkPrivKey (#1759) 2026-04-02 15:22:44 +00:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> 0741583f78 agent: read queues in batches for subscriptions (#1758)
* agent: read queues in batches for subscriptions

* resubscribe in batches too

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-04-01 16:07:17 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> f8f172f32f agent: fix race when pending subscriptions are never subscribed (#1756)
* agent: fix race when pending subscriptions are never subscribed

* small agent

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-31 19:16:54 +01:00
spaced4ndy 9c07ddff3c agent: allow to use existing connId for getConnShortLinkAsync (#1752) 2026-03-30 09:48:31 +00:00
Evgeny Poberezkin 50b71d3e56 6.5.0.12 2026-03-29 07:54:48 +01:00
EvgenyandEvgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> a1b762992b agent: pass key and link ID when preparing group link (#1754)
* agent: pass key and link ID when preparing group link

* binding

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-28 20:29:48 +00:00
Evgeny Poberezkin 1e0093be9a docs: update whitepaper 2026-03-26 20:48:47 +00:00
sh 1a12ee0a5a xftp-web: version bump to 0.3.0 (#1742) 2026-03-20 11:43:43 +00:00
sh efcef2d1fd xftp-web: add postgres schema cleanup for integration tests (#1741)
Stale postgres schema leaked pending XFTP operations between
cross-language tests, causing N-1 of N tests to fail.
2026-03-20 09:58:56 +00:00
224 changed files with 19147 additions and 3775 deletions
+26 -12
View File
@@ -24,6 +24,8 @@ jobs:
- name: Clone project
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/checkout@v3
with:
submodules: recursive
- name: Build changelog
id: build_changelog
@@ -114,6 +116,8 @@ jobs:
- name: Clone project
if: matrix.should_run == true
uses: actions/checkout@v3
with:
submodules: recursive
- name: Set up Docker Buildx
if: matrix.should_run == true
@@ -173,7 +177,7 @@ jobs:
-v ${{ github.workspace }}:/project \
build/${{ matrix.os }}:latest
- name: Build smp-server (postgresql) and tests
- name: Build smp-server, xftp-server (postgresql) and tests
if: matrix.should_run == true
shell: docker exec -t builder sh -eu {0}
run: |
@@ -182,12 +186,12 @@ jobs:
cabal update
cabal build --jobs=$(nproc) --enable-tests -fserver_postgres
mkdir -p /out
for i in smp-server simplexmq-test; do
for i in smp-server xftp-server simplexmq-test; do
bin=$(find /project/dist-newstyle -name "$i" -type f -executable)
chmod +x "$bin"
mv "$bin" /out/
done
strip /out/smp-server
strip /out/smp-server /out/xftp-server
- name: Copy simplexmq-test from container
if: matrix.should_run == true
@@ -195,19 +199,29 @@ jobs:
run: |
docker cp builder:/out/simplexmq-test .
- name: Copy smp-server (postgresql) from container and prepare it
- name: Copy smp-server, xftp-server (postgresql) from container and prepare it
if: startsWith(github.ref, 'refs/tags/v') && matrix.should_run == true
id: prepare-postgres
shell: bash
run: |
name="smp-server-postgres-ubuntu-${{ matrix.os_underscore }}-${{ matrix.arch }}"
docker cp builder:/out/smp-server $name
printf 'bins<<EOF\n' > bins.output
printf 'hashes<<EOF\n' > hashes.output
path="${{ github.workspace }}/$name"
echo "bin=$path" >> $GITHUB_OUTPUT
for i in smp-server xftp-server; do
name="${i}-postgres-ubuntu-${{ matrix.os_underscore }}-${{ matrix.arch }}"
docker cp builder:/out/$i $name
hash="SHA2-256($name)= $(openssl sha256 $path | cut -d' ' -f 2)"
printf 'hash=%s' "$hash" >> $GITHUB_OUTPUT
path="${{ github.workspace }}/$name"
hash="SHA2-256($name)= $(openssl sha256 $path | cut -d' ' -f 2)"
printf '%s\n' "$path" >> bins.output
printf '%s\n\n' "$hash" >> hashes.output
done
printf 'EOF\n' >> bins.output
printf 'EOF\n' >> hashes.output
cat bins.output >> "$GITHUB_OUTPUT"
cat hashes.output >> "$GITHUB_OUTPUT"
- name: Build everything else (standard)
if: matrix.should_run == true
@@ -257,10 +271,10 @@ jobs:
fail_on_unmatched_files: true
body: |
${{ steps.prepare-regular.outputs.hashes }}
${{ steps.prepare-postgres.outputs.hash }}
${{ steps.prepare-postgres.outputs.hashes }}
files: |
${{ steps.prepare-regular.outputs.bins }}
${{ steps.prepare-postgres.outputs.bin }}
${{ steps.prepare-postgres.outputs.bins }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+2
View File
@@ -20,6 +20,8 @@ jobs:
steps:
- name: Clone project
uses: actions/checkout@v4
with:
submodules: recursive
- name: Log in to Docker Hub
uses: simplex-chat/docker-login-action@v3
+2
View File
@@ -11,6 +11,8 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
submodules: recursive
- name: Get latest release
shell: bash
+6
View File
@@ -0,0 +1,6 @@
[submodule "cbits/libbbs"]
path = cbits/libbbs
url = https://github.com/simplex-chat/libbbs.git
[submodule "cbits/blst"]
path = cbits/blst
url = https://github.com/supranational/blst.git
+36
View File
@@ -1,3 +1,39 @@
# 6.5.1
Version 6.5.1.0
XFTP client:
- backwards compatible file header decoding.
# 6.5.0
Version 6.5.0.17
SMP agent:
- improve subscriptions
- reduce memory usage and retries during initial subscription (#1758)
- fix race resulting in pending subscriptions never subscribed (#1756)
- batch processing of subscription results and errors (#1652)
- reduce memory usage of active subscriptions.
- drop message after N reception attempts (#1762)
- fix possible deadlocks of queue overloading when processing messages (#1713)
- improved APIs for short link management and creation.
- support multiple link owners in link data (#1701)
SMP server:
- store messages in PostgreSQL (#1622).
- reduce memory usage with PostgreSQL database - do not use queue cache (#1637)
- fix in-memory server not restoring queue/service associations after 2+ restarts (#1618)
XFTP server:
- support PostgreSQL database.
- add server page.
- support uploads from web clients.
Servers:
- better socket leak prevention during TLS handshake, NetworkError type to bette diagnose connection errors (#1619)
- use "=" as default INI key-value separator (#1767)
# 6.4.4
Servers:
+1 -1
View File
@@ -33,7 +33,7 @@ To initialize the server use `smp-server init -n <fqdn>` (or `smp-server init --
SMP server uses in-memory persistence with an optional append-only log of created queues that allows to re-start the server without losing the connections. This log is compacted on every server restart, permanently removing suspended and removed queues.
To enable store log, initialize server using `smp-server -l` command, or modify `smp-server.ini` created during initialization (uncomment `enable: on` option in the store log section). Use `smp-server --help` for other usage tips.
To enable store log, initialize server using `smp-server -l` command, or modify `smp-server.ini` created during initialization (uncomment `enable = on` option in the store log section). Use `smp-server --help` for other usage tips.
Starting from version 2.3.0, when store log is enabled, the server would also enable saving undelivered messages on exit and restoring them on start. This can be disabled via a separate setting `restore_messages` in `smp-server.ini` file. Saving messages would only work if the server is stopped with SIGINT signal (keyboard interrupt), if it is stopped with SIGTERM signal the messages would not be saved.
+2 -2
View File
@@ -105,13 +105,13 @@
class="text-[16px] leading-[26px] tracking-[0.01em] nav-link-text text-black dark:text-white before:bg-black dark:before:bg-white">Server
information</span></a>
</li>
<x-xftpConfig>
<!-- <x-xftpConfig>
<li class="nav-link relative"><a href="/file"
class="flex items-center justify-between gap-2 lg:py-5 whitespace-nowrap"><span
class="text-[16px] leading-[26px] tracking-[0.01em] nav-link-text text-black dark:text-white before:bg-black dark:before:bg-white">File
transfer</span></a>
</li>
</x-xftpConfig>
</x-xftpConfig> -->
</ul><a target="_blank" href="https://github.com/simplex-chat/simplex-chat#help-us-with-donations"
class="whitespace-nowrap flex items-center gap-1 self-center text-white dark:text-black text-[16px] font-medium tracking-[0.02em] rounded-[34px] bg-primary-light dark:bg-primary-dark py-3 lg:py-2 px-20 lg:px-5 mb-16 lg:mb-0">Donate</a>
</div>
+3 -3
View File
@@ -34,7 +34,7 @@ xftpMediaContent = $(embedDir "apps/xftp-server/static/media/")
xftpFilePageHtml :: ByteString
xftpFilePageHtml = $(embedFile "apps/xftp-server/static/file.html")
xftpGenerateSite :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> FilePath -> IO ()
xftpGenerateSite :: XFTPServerConfig s -> Maybe ServerPublicInfo -> Maybe TransportHost -> FilePath -> IO ()
xftpGenerateSite cfg info onionHost path = do
let substs = xftpSubsts cfg info onionHost
Web.generateSite embeddedContent (render (Web.indexHtml embeddedContent) substs) [] path
@@ -50,10 +50,10 @@ xftpGenerateSite cfg info onionHost path = do
createDirectoryIfMissing True dir
forM_ content_ $ \(fp, content) -> B.writeFile (dir </> fp) content
xftpServerInformation :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> ByteString
xftpServerInformation :: XFTPServerConfig s -> Maybe ServerPublicInfo -> Maybe TransportHost -> ByteString
xftpServerInformation cfg info onionHost = render (Web.indexHtml embeddedContent) (xftpSubsts cfg info onionHost)
xftpSubsts :: XFTPServerConfig -> Maybe ServerPublicInfo -> Maybe TransportHost -> [(ByteString, Maybe ByteString)]
xftpSubsts :: XFTPServerConfig s -> Maybe ServerPublicInfo -> Maybe TransportHost -> [(ByteString, Maybe ByteString)]
xftpSubsts XFTPServerConfig {fileExpiration, logStatsInterval, allowNewFiles, newFileBasicAuth} information onionHost =
[("smpConfig", Nothing), ("xftpConfig", Just "y")] <> substConfig <> serverInfoSubsts simplexmqSource information <> [("onionHost", strEncode <$> onionHost), ("iniFileName", Just "file-server.ini")]
where
@@ -619,7 +619,6 @@ async function handleEncrypt(id, data, fileName) {
const digest = sha512Streaming([encData], (done) => {
self.postMessage({ id, type: "progress", done: source.length + done, total });
}, encDataLen);
console.log(`[WORKER-DBG] encrypt: encData.len=${encData.length} digest=${_whex(digest, 64)} chunkSizes=[${chunkSizes.join(",")}]`);
const dir = await getSessionDir();
const fileHandle = await dir.getFileHandle("upload.bin", { create: true });
const writeHandle = await fileHandle.createSyncAccessHandle();
@@ -643,9 +642,7 @@ function handleReadChunk(id, offset, size) {
}
async function handleDecryptAndStore(id, dhSecret, nonce, body, chunkDigest, chunkNo) {
const bodyArr = new Uint8Array(body);
console.log(`[WORKER-DBG] store chunk=${chunkNo} body.len=${bodyArr.length} nonce=${_whex(nonce, 24)} dhSecret=${_whex(dhSecret)} digest=${_whex(chunkDigest, 32)} body[0..8]=${_whex(bodyArr)} body[-8..]=${_whex(bodyArr.slice(-8))}`);
const decrypted = decryptReceivedChunk(dhSecret, nonce, bodyArr, chunkDigest);
console.log(`[WORKER-DBG] decrypted chunk=${chunkNo} len=${decrypted.length} [0..8]=${_whex(decrypted)} [-8..]=${_whex(decrypted.slice(-8))}`);
if (useMemory) {
memoryChunks.set(chunkNo, decrypted);
self.postMessage({ id, type: "stored" });
@@ -660,7 +657,6 @@ async function handleDecryptAndStore(id, dhSecret, nonce, body, chunkDigest, chu
currentDownloadOffset += decrypted.length;
chunkMeta.set(chunkNo, { offset, size: decrypted.length });
const written = downloadWriteHandle.write(decrypted, { at: offset });
console.log(`[WORKER-DBG] OPFS write chunk=${chunkNo} offset=${offset} size=${decrypted.length} written=${written}`);
if (written !== decrypted.length) {
console.warn(`[WORKER] OPFS write failed chunk=${chunkNo}: ${written}/${decrypted.length}, falling back to in-memory storage`);
for (const [cn, meta] of chunkMeta.entries()) {
@@ -684,23 +680,16 @@ async function handleDecryptAndStore(id, dhSecret, nonce, body, chunkDigest, chu
return;
}
downloadWriteHandle.flush();
const verifyBuf = new Uint8Array(Math.min(8, decrypted.length));
downloadWriteHandle.read(verifyBuf, { at: offset });
const verifyEnd = new Uint8Array(Math.min(8, decrypted.length));
downloadWriteHandle.read(verifyEnd, { at: offset + decrypted.length - verifyEnd.length });
console.log(`[WORKER-DBG] OPFS verify chunk=${chunkNo} readBack[0..8]=${_whex(verifyBuf)} readBack[-8..]=${_whex(verifyEnd)} expected[0..8]=${_whex(decrypted)} expected[-8..]=${_whex(decrypted.slice(-8))}`);
self.postMessage({ id, type: "stored" });
}
async function handleVerifyAndDecrypt(id, size, digest, key, nonce) {
console.log(`[WORKER-DBG] verify: expectedSize=${size} expectedDigest=${_whex(digest, 64)} useMemory=${useMemory} chunkMeta.size=${chunkMeta.size} memoryChunks.size=${memoryChunks.size}`);
const chunks = [];
let totalSize = 0;
const total = size * 3;
let done = 0;
if (useMemory) {
const sorted = [...memoryChunks.entries()].sort((a, b) => a[0] - b[0]);
for (const [chunkNo, data] of sorted) {
console.log(`[WORKER-DBG] verify memory chunk=${chunkNo} size=${data.length}`);
for (const [, data] of sorted) {
chunks.push(data);
totalSize += data.length;
done += data.length;
@@ -715,12 +704,10 @@ async function handleVerifyAndDecrypt(id, size, digest, key, nonce) {
const dir = await getSessionDir();
const fileHandle = await dir.getFileHandle("download.bin");
const readHandle = await fileHandle.createSyncAccessHandle();
console.log(`[WORKER-DBG] verify: OPFS file size=${readHandle.getSize()}`);
const sortedEntries = [...chunkMeta.entries()].sort((a, b) => a[0] - b[0]);
for (const [chunkNo, meta] of sortedEntries) {
for (const [, meta] of sortedEntries) {
const buf = new Uint8Array(meta.size);
const bytesRead = readHandle.read(buf, { at: meta.offset });
console.log(`[WORKER-DBG] verify read chunk=${chunkNo} offset=${meta.offset} size=${meta.size} bytesRead=${bytesRead} [0..8]=${_whex(buf)} [-8..]=${_whex(buf.slice(-8))}`);
readHandle.read(buf, { at: meta.offset });
chunks.push(buf);
totalSize += meta.size;
done += meta.size;
@@ -745,20 +732,9 @@ async function handleVerifyAndDecrypt(id, size, digest, key, nonce) {
}
const actualDigest = r.crypto_hash_sha512_final(state);
if (!digestEqual(actualDigest, digest)) {
console.error(`[WORKER-DBG] DIGEST MISMATCH: expected=${_whex(digest, 64)} actual=${_whex(actualDigest, 64)} chunks=${chunks.length} totalSize=${totalSize}`);
const state2 = r.crypto_hash_sha512_init();
for (let i = 0; i < chunks.length; i++) {
const chunk = chunks[i];
for (let off = 0; off < chunk.length; off += hashSEG) {
r.crypto_hash_sha512_update(state2, chunk.subarray(off, Math.min(off + hashSEG, chunk.length)));
}
const chunkDigest = sha512Streaming([chunk]);
console.error(`[WORKER-DBG] chunk[${i}] size=${chunk.length} sha512=${_whex(chunkDigest, 32)}… [0..8]=${_whex(chunk)} [-8..]=${_whex(chunk.slice(-8))}`);
}
self.postMessage({ id, type: "error", message: "File digest mismatch" });
return;
}
console.log(`[WORKER-DBG] verify: digest OK`);
const result = decryptChunks(BigInt(size), chunks, key, nonce, (d) => {
self.postMessage({ id, type: "progress", done: size * 2 + d, total });
});
@@ -334,11 +334,6 @@ class WorkerBackend {
const nonceCopy = new Uint8Array(nonce);
const digestCopy = new Uint8Array(digest);
const buf = this.toTransferable(body);
const hex = (b, n = 8) => {
const u = b instanceof ArrayBuffer ? new Uint8Array(b) : b;
return Array.from(u.slice(0, n)).map((x) => x.toString(16).padStart(2, "0")).join("");
};
console.log(`[BACKEND-DBG] chunk=${chunkNo} body.len=${body.length} body.byteOff=${body.byteOffset} buf.byteLen=${buf.byteLength} nonce=${hex(nonceCopy, 24)} dhSecret=${hex(dhSecretCopy)} digest=${hex(digestCopy, 32)} buf[0..8]=${hex(buf)} body[-8..]=${hex(body.slice(-8))}`);
await this.send(
{ type: "decryptAndStoreChunk", dhSecret: dhSecretCopy, nonce: nonceCopy, body: buf, chunkDigest: digestCopy, chunkNo },
[buf]
@@ -10913,14 +10908,12 @@ async function sendXFTPCommandOnce(client, privateKey, entityId, cmdBytes, chunk
const block = encodeAuthTransmission(client.sessionId, corrId, entityId, cmdBytes, privateKey);
const reqBody = chunkData ? concatBytes$1(block, chunkData) : block;
const fullResp = await client.transport.post(reqBody);
console.log(`[XFTP-DBG] sendOnce: fullResp.length=${fullResp.length} entityId=${_hex(entityId)} cmdTag=${cmdBytes[0]}`);
if (fullResp.length < XFTP_BLOCK_SIZE) {
console.error("[XFTP] Response too short: %d bytes (expected >= %d)", fullResp.length, XFTP_BLOCK_SIZE);
throw new Error("Server response too short");
}
const respBlock = fullResp.subarray(0, XFTP_BLOCK_SIZE);
const body = fullResp.subarray(XFTP_BLOCK_SIZE);
console.log(`[XFTP-DBG] sendOnce: body.length=${body.length} body.byteOffset=${body.byteOffset} body.buffer.byteLength=${body.buffer.byteLength}`);
const raw = blockUnpad(respBlock);
if (raw.length < 20) {
const text = new TextDecoder().decode(raw);
@@ -10939,18 +10932,13 @@ async function sendXFTPCommandOnce(client, privateKey, entityId, cmdBytes, chunk
}
return { response, body };
}
function _hex(b, n = 8) {
return Array.from(b.slice(0, n)).map((x) => x.toString(16).padStart(2, "0")).join("");
}
async function sendXFTPCommand(agent, server, privateKey, entityId, cmdBytes, chunkData, maxRetries = 3) {
let clientP = getXFTPServerClient(agent, server);
let client = await clientP;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
if (attempt > 1) console.log(`[XFTP-DBG] sendCmd: retry attempt=${attempt}/${maxRetries}`);
return await sendXFTPCommandOnce(client, privateKey, entityId, cmdBytes, chunkData);
} catch (e) {
console.log(`[XFTP-DBG] sendCmd: attempt=${attempt} failed: ${e instanceof Error ? e.message : String(e)} retriable=${isRetriable(e)}`);
if (!isRetriable(e)) {
throw categorizeError(e);
}
@@ -10979,7 +10967,6 @@ async function downloadXFTPChunkRaw(agent, server, rpKey, fId) {
const { response, body } = await sendXFTPCommand(agent, server, rpKey, fId, cmd);
if (response.type !== "FRFile") throw new Error("unexpected response: " + response.type);
const dhSecret = dh(response.rcvDhKey, privateKey);
console.log(`[XFTP-DBG] dlChunkRaw: body.length=${body.length} nonce=${_hex(response.nonce, 24)} dhSecret=${_hex(dhSecret)} body[0..8]=${_hex(body)} body[-8..]=${_hex(body.slice(-8))}`);
return { dhSecret, nonce: response.nonce, body };
}
async function downloadXFTPChunk(agent, server, rpKey, fId, digest) {
@@ -11012,7 +10999,6 @@ function encryptFileForUpload(source, fileName) {
const encSize = BigInt(chunkSizes.reduce((a, b) => a + b, 0));
const encData = encryptFile(source, fileHdr, key, nonce, fileSize, encSize);
const digest = sha512Streaming([encData]);
console.log(`[AGENT-DBG] encrypt: encData.len=${encData.length} digest=${_dbgHex(digest, 64)} chunkSizes=[${chunkSizes.join(",")}]`);
return { encData, digest, key, nonce, chunkSizes };
}
const DEFAULT_REDIRECT_THRESHOLD = 400;
@@ -11169,9 +11155,7 @@ async function downloadFileRaw(agent, fd, onRawChunk, options) {
if (err) throw new Error("downloadFileRaw: " + err);
const { onProgress} = options ?? {};
if (fd.redirect !== null) {
console.log(`[AGENT-DBG] resolving redirect: outer size=${fd.size} chunks=${fd.chunks.length}`);
fd = await resolveRedirect(agent, fd);
console.log(`[AGENT-DBG] resolved: size=${fd.size} chunks=${fd.chunks.length} digest=${Array.from(fd.digest.slice(0, 16)).map((x) => x.toString(16).padStart(2, "0")).join("")}`);
}
const resolvedFd = fd;
let downloaded = 0;
@@ -11189,7 +11173,6 @@ async function downloadFileRaw(agent, fd, onRawChunk, options) {
const seed = decodePrivKeyEd25519(replica.replicaKey);
const kp = ed25519KeyPairFromSeed(seed);
const raw = await downloadXFTPChunkRaw(agent, server, kp.privateKey, replica.replicaId);
console.log(`[AGENT-DBG] chunk=${chunk.chunkNo} body.len=${raw.body.length} expectedChunkSize=${chunk.chunkSize} digest=${_dbgHex(chunk.digest, 32)} body.byteOffset=${raw.body.byteOffset} body.buffer.byteLength=${raw.body.buffer.byteLength}`);
await onRawChunk({
chunkNo: chunk.chunkNo,
dhSecret: raw.dhSecret,
Submodule
+1
Submodule cbits/blst added at db3defd0d5
+24
View File
@@ -0,0 +1,24 @@
// SPDX-License-Identifier: Apache-2.0
// getentropy() shim for Windows, where it is absent from the CRT.
// Follows the POSIX contract: fills `buffer` with `length` random bytes
// (length must not exceed 256), returns 0 on success or -1 with errno set.
#ifdef _WIN32
#include <errno.h>
#include <stddef.h>
#include <windows.h>
#include <bcrypt.h>
int getentropy(void *buffer, size_t length) {
if (length > 256) {
errno = EIO;
return -1;
}
NTSTATUS status = BCryptGenRandom(NULL, (PUCHAR)buffer, (ULONG)length,
BCRYPT_USE_SYSTEM_PREFERRED_RNG);
if (!BCRYPT_SUCCESS(status)) {
errno = EIO;
return -1;
}
return 0;
}
#endif
Submodule
+1
Submodule cbits/libbbs added at 59a0f4bf32
@@ -0,0 +1,472 @@
# XFTP Server PostgreSQL Backend
## Overview
Add PostgreSQL backend support to xftp-server, following the SMP server pattern. Supports bidirectional migration between STM (in-memory with StoreLog) and PostgreSQL backends.
## Goals
- PostgreSQL-backed file metadata storage as an alternative to STM + StoreLog
- Polymorphic server code via `FileStoreClass` typeclass with IO-based methods (following `QueueStoreClass` pattern)
- Bidirectional migration: StoreLog <-> PostgreSQL via CLI commands
- Shared `server_postgres` cabal flag (same flag enables both SMP and XFTP Postgres support)
- INI-based backend selection at runtime
## Architecture
### FileStoreClass Typeclass
IO-based typeclass following the `QueueStoreClass` pattern — each method is a self-contained IO action, with the implementation responsible for its own atomicity (STM backend wraps in `atomically`, Postgres backend uses database transactions):
```haskell
class FileStoreClass s where
type FileStoreConfig s
-- Lifecycle
newFileStore :: FileStoreConfig s -> IO s
closeFileStore :: s -> IO ()
-- File operations
addFile :: s -> SenderId -> FileInfo -> RoundedFileTime -> ServerEntityStatus -> IO (Either XFTPErrorType ())
setFilePath :: s -> SenderId -> FilePath -> IO (Either XFTPErrorType ())
addRecipient :: s -> SenderId -> FileRecipient -> IO (Either XFTPErrorType ())
getFile :: s -> SFileParty p -> XFTPFileId -> IO (Either XFTPErrorType (FileRec, C.APublicAuthKey))
deleteFile :: s -> SenderId -> IO (Either XFTPErrorType ())
blockFile :: s -> SenderId -> BlockingInfo -> Bool -> IO (Either XFTPErrorType ())
deleteRecipient :: s -> RecipientId -> FileRec -> IO ()
ackFile :: s -> RecipientId -> IO (Either XFTPErrorType ())
-- Expiration (with LIMIT for Postgres; called in a loop until empty)
expiredFiles :: s -> Int64 -> Int -> IO [(SenderId, Maybe FilePath, Word32)]
-- Storage and stats (for init-time computation)
getUsedStorage :: s -> IO Int64
getFileCount :: s -> IO Int
```
- STM backend: each method wraps its STM transaction in `atomically` internally.
- Postgres backend: each method runs its query via `withDB` / database connection internally.
No polymorphic monad or `runStore` dispatcher needed — unlike `MsgStoreClass`, XFTP file operations are individually atomic and don't require grouping multiple operations into backend-dependent transactions.
### PostgresFileStore Data Type
```haskell
data PostgresFileStore = PostgresFileStore
{ dbStore :: DBStore,
dbStoreLog :: Maybe (StoreLog 'WriteMode)
}
```
- `dbStore` — connection pool created via `createDBStore`, runs schema migrations on init.
- `dbStoreLog` — optional parallel log file (enabled by `db_store_log` INI setting). When present, every mutation (`addFile`, `setFilePath`, `deleteFile`, `blockFile`, `addRecipient`, `ackFile`) also writes to this log via a `withLog` wrapper. `withLog` is called AFTER the DB operation succeeds (so the log reflects committed state only). Log write failures are non-fatal (logged as warnings, do not fail the DB operation). This provides an audit trail and enables recovery via export.
`closeFileStore` for Postgres calls `closeDBStore` (closes connection pool) then `mapM_ closeStoreLog dbStoreLog` (flushes and closes the parallel log). For STM, it closes the storeLog. Called from a `finally` block during server shutdown, matching SMP's `stopServer``closeMsgStore``closeQueueStore` pattern.
### STMFileStore Type
After extracting from current `Store.hs`, `STMFileStore` retains the file and recipient maps but no longer owns `usedStorage` (moved to `XFTPEnv`):
```haskell
data STMFileStore = STMFileStore
{ files :: TMap SenderId FileRec,
recipients :: TMap RecipientId (SenderId, RcvPublicAuthKey)
}
```
`closeFileStore` for STM is a no-op (TMaps are garbage-collected; the env-level `storeLog` is closed separately by the server).
### Error Handling
Postgres operations follow SMP's `withDB` / `handleDuplicate` pattern:
```haskell
withDB :: Text -> PostgresFileStore -> (DB.Connection -> IO (Either XFTPErrorType a)) -> ExceptT XFTPErrorType IO a
withDB op st action =
ExceptT $ E.try (withTransaction (dbStore st) action) >>= either logErr pure
where
logErr :: E.SomeException -> IO (Either XFTPErrorType a)
logErr e = logError ("STORE: " <> err) $> Left INTERNAL
where
err = op <> ", withDB, " <> tshow e
handleDuplicate :: SqlError -> IO (Either XFTPErrorType a)
handleDuplicate e = case constraintViolation e of
Just (UniqueViolation _) -> pure $ Left DUPLICATE_
_ -> E.throwIO e
```
- All DB operations wrapped in `withDB` — catches exceptions, logs, returns `INTERNAL`.
- Unique constraint violations caught by `handleDuplicate` and mapped to `DUPLICATE_`.
- UPDATE operations verified with `assertUpdated` — returns `AUTH` if 0 rows affected (matching SMP pattern, prevents silent failures when WHERE clause doesn't match).
- Critical sections (DB write + TVar update) wrapped in `uninterruptibleMask_` to prevent async exceptions from leaving inconsistent state between DB and TVars.
### FileRec and TVar Fields
`FileRec` retains its `TVar` fields (matching SMP's `PostgresQueue` pattern):
```haskell
data FileRec = FileRec
{ senderId :: SenderId,
fileInfo :: FileInfo,
filePath :: TVar (Maybe FilePath),
recipientIds :: TVar (Set RecipientId),
createdAt :: RoundedFileTime,
fileStatus :: TVar ServerEntityStatus
}
```
- **STM backend**: TVars are the source of truth, as currently.
- **Postgres backend**: `getFile` reads from DB and creates a `FileRec` with fresh TVars populated from the DB row (matching SMP's `mkQ` pattern — `newTVarIO` per load). Mutation methods (`setFilePath`, `blockFile`, etc.) update both the DB (persistence) and the TVars (in-session consistency). The `recipientIds` TVar is initialized to `S.empty` — no subquery needed because no server code reads `recipientIds` directly; all recipient operations go through the typeclass methods (`addRecipient`, `deleteRecipient`, `ackFile`), which query the `recipients` table for Postgres.
### usedStorage Ownership
`usedStorage :: TVar Int64` moves from the store to `XFTPEnv`. The store typeclass does **not** manage `usedStorage` — it only provides `getUsedStorage` for init-time computation.
- **STM init**: StoreLog replay calls `setFilePath` (which only sets the filePath TVar — the STM `setFilePath` implementation is changed to **not** update `usedStorage`). Similarly, STM `deleteFile` (Store.hs line 117) and `blockFile` (line 125) are changed to **not** update `usedStorage` — the server handles all `usedStorage` adjustments externally. After replay, `getUsedStorage` computes the sum over all file sizes (matching current `countUsedStorage` behavior).
- **Postgres init**: `getUsedStorage` executes `SELECT COALESCE(SUM(file_size), 0) FROM files`.
- **Runtime**: Server manages `usedStorage` TVar directly for reserve/commit/rollback during uploads, and adjusts after `deleteFile`/`blockFile` calls.
**Note on `getUsedStorage` semantics**: The current STM `countUsedStorage` sums all file sizes unconditionally (including files without `filePath` set, i.e., created but not yet uploaded). The Postgres `getUsedStorage` matches this: `SELECT SUM(file_size) FROM files` (no `WHERE file_path IS NOT NULL`). In practice, orphaned files (created but never uploaded) are rare and short-lived (expired within 48h), so the difference is negligible. A future improvement could filter by `file_path IS NOT NULL` in both backends to reflect actual disk usage more accurately.
### Server.hs Refactoring
`Server.hs` becomes polymorphic over `FileStoreClass s`. Since all typeclass methods are IO, call sites replace `atomically` with direct IO calls to the store.
**Call sites requiring changes** (exhaustive list):
1. **`receiveServerFile`** (line 563): `atomically $ writeTVar filePath (Just fPath)``setFilePath store senderId fPath`. The `reserve` logic (line 551-555) stays as direct TVar manipulation on `usedStorage` from `XFTPEnv`.
2. **`verifyXFTPTransmission`** (line 453): `atomically $ verify =<< getFile st party fId` — the `getFile` call and subsequent `readTVar fileStatus` are in a single `atomically` block. Refactored to: `getFile st party fId` (IO), then `readTVarIO (fileStatus fr)` from the returned `FileRec` (safe for both backends — STM TVar is the source of truth, Postgres TVar is a fresh snapshot from DB).
3. **`retryAdd`** (line 516): Signature `XFTPFileId -> STM (Either XFTPErrorType a)``XFTPFileId -> IO (Either XFTPErrorType a)`. The `atomically` call (line 520) replaced with `liftIO`.
4. **`deleteOrBlockServerFile_`** (line 620): Parameter `FileStore -> STM (Either XFTPErrorType ())``FileStoreClass s => s -> IO (Either XFTPErrorType ())`. The `atomically` call (line 626) removed — the store method is already IO. After the store action, server adjusts `usedStorage` TVar in `XFTPEnv` based on `fileInfo.size`.
5. **`ackFileReception`** (line 605): `atomically $ deleteRecipient st rId fr``deleteRecipient st rId fr`.
6. **Control port `CPDelete`/`CPBlock`** (lines 371, 377): `atomically $ getFile fs SFRecipient fileId``getFile fs SFRecipient fileId`.
7. **`expireServerFiles`** (line 636): Replace per-file `expiredFilePath` iteration with batched `expiredFiles st old batchSize`, which returns `[(SenderId, Maybe FilePath, Word32)]` — the `Word32` file size is needed so the server can adjust the `usedStorage` TVar after each deletion. Called in a loop until the returned list is empty. The `itemDelay` between files applies to the deletion loop over each batch, not the query itself. STM backend ignores the batch size limit (returns all expired files from TMap scan); Postgres uses `LIMIT`.
8. **`restoreServerStats`** (line 694): `FileStore {files, usedStorage} <- asks store` accesses store fields directly. Refactored to: `usedStorage` from `XFTPEnv` via `asks usedStorage`, file count via `getFileCount store`. STM: `M.size <$> readTVarIO files`. Postgres: `SELECT COUNT(*) FROM files`.
### Store Config Selection
GADT in `Env.hs`:
```haskell
data XFTPStoreConfig s where
XSCMemory :: Maybe FilePath -> XFTPStoreConfig STMFileStore
#if defined(dbServerPostgres)
XSCDatabase :: PostgresFileStoreCfg -> XFTPStoreConfig PostgresFileStore
#endif
```
`XFTPEnv` becomes polymorphic:
```haskell
data XFTPEnv s = XFTPEnv
{ config :: XFTPServerConfig,
store :: s,
usedStorage :: TVar Int64,
storeLog :: Maybe (StoreLog 'WriteMode),
...
}
```
The `M` monad (`ReaderT (XFTPEnv s) IO`) and all functions in `Server.hs` gain `FileStoreClass s =>` constraints.
**StoreLog lifecycle per backend:**
- **STM mode**: `storeLog = Just sl` (current behavior — append-only log for persistence and recovery).
- **Postgres mode**: `storeLog = Nothing` (main storeLog disabled — Postgres is the source of truth). The optional parallel `dbStoreLog` inside `PostgresFileStore` provides audit/recovery if enabled via `db_store_log` INI setting.
The existing `withFileLog` pattern in Server.hs continues to work unchanged — it maps over `Maybe (StoreLog 'WriteMode)`, which is `Nothing` in Postgres mode so the calls become no-ops.
### Main.hs Store Type Dispatch
The `Start` CLI command gains a `--confirm-migrations` flag (default `MCConsole` — manual prompt, matching SMP's `StartOptions`). For automated deployments, `--confirm-migrations up` auto-applies forward migrations. The import command uses `MCYesUp` (always auto-apply).
Following SMP's existential dispatch pattern (`AStoreType` + `run`), `Main.hs` selects the store type from INI config and dispatches to the polymorphic server:
```haskell
runServer ini = do
let storeType = fromRight "memory" $ lookupValue "STORE_LOG" "store_files" ini
case storeType of
"memory" -> run $ XSCMemory (enableStoreLog $> storeLogFilePath)
"database" ->
#if defined(dbServerPostgres)
run $ XSCDatabase PostgresFileStoreCfg {..}
#else
exitError "server not compiled with Postgres support"
#endif
_ -> exitError $ "Invalid store_files value: " <> storeType
where
run :: FileStoreClass s => XFTPStoreConfig s -> IO ()
run storeCfg = do
env <- newXFTPServerEnv storeCfg config
runReaderT (xftpServer config) env
```
**`newXFTPServerEnv` refactored signature:**
```haskell
newXFTPServerEnv :: FileStoreClass s => XFTPStoreConfig s -> XFTPServerConfig -> IO (XFTPEnv s)
newXFTPServerEnv storeCfg config = do
(store, storeLog) <- case storeCfg of
XSCMemory storeLogPath -> do
st <- newFileStore ()
sl <- mapM (`readWriteFileStore` st) storeLogPath
pure (st, sl)
XSCDatabase dbCfg -> do
st <- newFileStore dbCfg
pure (st, Nothing) -- main storeLog disabled for Postgres
usedStorage <- newTVarIO =<< getUsedStorage store
...
pure XFTPEnv {config, store, usedStorage, storeLog, ...}
```
### Startup Config Validation
Following SMP's `checkMsgStoreMode` pattern, `Main.hs` validates config before starting:
- **`store_files=database` + StoreLog file exists** (without `db_store_log=on`): Error — "StoreLog file present but store_files is `database`. Use `xftp-server database import` to migrate, or set `db_store_log: on`."
- **`store_files=database` + schema doesn't exist**: Error — "Create schema in PostgreSQL or use `xftp-server database import`."
- **`store_files=memory` + Postgres schema exists**: Warning — "Postgres schema exists but store_files is `memory`. Data in Postgres will not be used."
- **Binary compiled without `server_postgres` + `store_files=database`**: Error — "Server not compiled with Postgres support."
## Module Structure
```
src/Simplex/FileTransfer/Server/
Store.hs -- FileStoreClass typeclass + shared types (FileRec, FileRecipient, etc.)
Store/
STM.hs -- STMFileStore (extracted from current Store.hs)
Postgres.hs -- PostgresFileStore [CPP-guarded]
Postgres/
Migrations.hs -- Schema migrations [CPP-guarded]
Config.hs -- PostgresFileStoreCfg [CPP-guarded]
StoreLog.hs -- Unchanged (interchange format for both backends + migration)
Env.hs -- XFTPStoreConfig GADT, polymorphic XFTPEnv
Main.hs -- Store selection, migration CLI commands
Server.hs -- Polymorphic over FileStoreClass
```
## PostgreSQL Schema
Initial migration (`20260325_initial`):
```sql
CREATE TABLE files (
sender_id BYTEA NOT NULL PRIMARY KEY,
file_size INT4 NOT NULL,
file_digest BYTEA NOT NULL,
sender_key BYTEA NOT NULL,
file_path TEXT,
created_at INT8 NOT NULL,
status TEXT NOT NULL DEFAULT 'active'
);
CREATE TABLE recipients (
recipient_id BYTEA NOT NULL PRIMARY KEY,
sender_id BYTEA NOT NULL REFERENCES files ON DELETE CASCADE,
recipient_key BYTEA NOT NULL
);
CREATE INDEX idx_recipients_sender_id ON recipients (sender_id);
CREATE INDEX idx_files_created_at ON files (created_at);
```
- `file_size` is `INT4` matching `Word32` in `FileInfo.size`
- `sender_key` and `recipient_key` stored as `BYTEA` using binary encoding via `C.encodePubKey` / `C.decodePubKey` (matching SMP's `ToField`/`FromField` instances for `APublicAuthKey` — includes algorithm type tag in the binary format)
- `file_path` nullable (set after upload completes via `setFilePath`)
- `ON DELETE CASCADE` for recipients when file is hard-deleted
- `created_at` stores rounded epoch seconds (1-hour precision, `RoundedFileTime`)
- `status` as TEXT via `StrEncoding` (`ServerEntityStatus`: `EntityActive`, `EntityBlocked info`, `EntityOff`)
- Hard deletes (no `deleted_at` column)
- No PL/pgSQL functions needed; `setFilePath` uses `WHERE file_path IS NULL` to prevent duplicate uploads (the `UPDATE` itself acquires a row-level lock)
- `used_storage` computed on startup: `SELECT COALESCE(SUM(file_size), 0) FROM files` (matches STM `countUsedStorage` — all files, see usedStorage Ownership section)
### Migrations Module
Following SMP's `QueueStore/Postgres/Migrations.hs` pattern:
```haskell
module Simplex.FileTransfer.Server.Store.Postgres.Migrations
( xftpServerMigrations,
)
where
import Data.List (sortOn)
import Data.Text (Text)
import Simplex.Messaging.Agent.Store.Shared
import Text.RawString.QQ (r)
xftpSchemaMigrations :: [(String, Text, Maybe Text)]
xftpSchemaMigrations =
[ ("20260325_initial", m20260325_initial, Nothing)
]
xftpServerMigrations :: [Migration]
xftpServerMigrations = sortOn name $ map migration xftpSchemaMigrations
where
migration (name, up, down) = Migration {name, up, down = down}
m20260325_initial :: Text
m20260325_initial =
[r|
CREATE TABLE files (
sender_id BYTEA NOT NULL PRIMARY KEY,
...
);
|]
```
The `Migration` type (from `Simplex.Messaging.Agent.Store.Shared`) has fields `{name :: String, up :: Text, down :: Maybe Text}`. Initial migration has `Nothing` for `down`. Future migrations should include `Just down_migration` for rollback support. Called via `createDBStore dbOpts xftpServerMigrations (MigrationConfig confirmMigrations Nothing)`.
### Postgres Operations
Key query patterns:
- **`addFile`**: `INSERT INTO files (...) VALUES (...)`, return `DUPLICATE_` on unique violation.
- **`setFilePath`**: `UPDATE files SET file_path = ? WHERE sender_id = ? AND file_path IS NULL`, verified with `assertUpdated` (returns `AUTH` if 0 rows affected — file not found or already uploaded). The `WHERE file_path IS NULL` prevents duplicate uploads; the `UPDATE` acquires a row lock implicitly. Only persists the path; `usedStorage` managed by server.
- **`addRecipient`**: `INSERT INTO recipients (...)`, plus check for duplicates. No need for `recipientIds` TVar update — Postgres derives it from the table.
- **`getFile`** (sender): `SELECT ... FROM files WHERE sender_id = ?`, returns auth key from `sender_key` column.
- **`getFile`** (recipient): `SELECT f.*, r.recipient_key FROM recipients r JOIN files f ON ... WHERE r.recipient_id = ?`.
- **`deleteFile`**: `DELETE FROM files WHERE sender_id = ?` (recipients cascade).
- **`blockFile`**: `UPDATE files SET status = ? WHERE sender_id = ?`. When `deleted = True`, the server adjusts `usedStorage` externally (matching current STM behavior where `blockFile` only updates status and storage, not `filePath`).
- **`expiredFiles`**: `SELECT sender_id, file_path, file_size FROM files WHERE created_at + ? < ? LIMIT ?` — batched query replaces per-file iteration, includes `file_size` for `usedStorage` adjustment. Called in a loop until no rows returned.
## INI Configuration
New keys in `[STORE_LOG]` section:
```ini
[STORE_LOG]
enable: on
store_files: memory # memory | database
db_connection: postgresql://xftp@/xftp_server_store
db_schema: xftp_server
db_pool_size: 10
db_store_log: off
expire_files_hours: 48
```
`store_files` selects the backend (`store_files` rather than `store_queues` because XFTP stores files, not queues):
- `memory` -> `XSCMemory` (current behavior)
- `database` -> `XSCDatabase` (requires `server_postgres` build flag)
### INI Template Generation (`xftp-server init`)
The `iniFileContent` function in `Main.hs` must be updated to generate the new keys in the `[STORE_LOG]` section. Following SMP's `iniDbOpts` pattern with `optDisabled'` (prefixes `"# "` when value equals default), Postgres keys are generated commented out by default:
```ini
[STORE_LOG]
enable: on
# File storage mode: `memory` or `database` (PostgreSQL).
store_files: memory
# Database connection settings for PostgreSQL database (`store_files: database`).
# db_connection: postgresql://xftp@/xftp_server_store
# db_schema: xftp_server
# db_pool_size: 10
# Write database changes to store log file
# db_store_log: off
expire_files_hours: 48
```
Reuses `iniDBOptions` from `Simplex.Messaging.Server.CLI` for runtime parsing (falls back to defaults when keys are commented out or missing). `enableDbStoreLog'` pattern (`settingIsOn "STORE_LOG" "db_store_log"`) controls `dbStoreLogPath`.
### PostgresFileStoreCfg
```haskell
data PostgresFileStoreCfg = PostgresFileStoreCfg
{ dbOpts :: DBOpts,
dbStoreLogPath :: Maybe FilePath,
confirmMigrations :: MigrationConfirmation
}
```
No `deletedTTL` (hard deletes).
### Default DB Options
```haskell
defaultXFTPDBOpts :: DBOpts
defaultXFTPDBOpts =
DBOpts
{ connstr = "postgresql://xftp@/xftp_server_store",
schema = "xftp_server",
poolSize = 10,
createSchema = False
}
```
## Migration CLI
Bidirectional migration via StoreLog as interchange format:
```
xftp-server database import [--database DB_CONN] [--schema DB_SCHEMA] [--pool-size N]
xftp-server database export [--database DB_CONN] [--schema DB_SCHEMA] [--pool-size N]
```
No `--table` flag needed (unlike SMP which has queues/messages/all) — XFTP has a single entity type (files + recipients, always migrated together).
CLI options reuse `dbOptsP` parser from `Simplex.Messaging.Server.CLI`.
### Import (StoreLog -> PostgreSQL)
1. Confirm: prompt user with database connection details and StoreLog path
2. Read and replay StoreLog into temporary `STMFileStore`
3. Connect to PostgreSQL, run schema migrations (`createSchema = True`, `confirmMigrations = MCYesUp`)
4. Batch-insert file records into `files` table using PostgreSQL COPY protocol (matching SMP's `batchInsertQueues` pattern for performance). Progress reported every 10k files.
5. Batch-insert recipient records into `recipients` table using COPY protocol
6. Verify counts: `SELECT COUNT(*) FROM files` / `recipients` — warn if mismatch
7. Rename StoreLog to `.bak` (prevents accidental re-import, preserves original for rollback)
8. Report counts
### Export (PostgreSQL -> StoreLog)
1. Confirm: prompt user with database connection details and output path. Fail if output file already exists.
2. Connect to PostgreSQL
3. Open new StoreLog file for writing
4. Fold over all file records, writing per file (in this order, matching existing `writeFileStore`): `AddFile` (with `ServerEntityStatus` — this preserves `EntityBlocked` state), `AddRecipients`, then `PutFile` (if `file_path` is set)
5. Report counts
Note: `AddFile` carries `ServerEntityStatus` which includes `EntityBlocked info`, so blocking state is preserved through export/import without needing separate `BlockFile` log entries.
File data on disk is untouched by migration — only metadata moves between backends.
## Cabal Integration
Shared `server_postgres` flag. New Postgres modules added to existing conditional block:
```cabal
if flag(server_postgres)
cpp-options: -DdbServerPostgres
exposed-modules:
...existing SMP modules...
Simplex.FileTransfer.Server.Store.Postgres
Simplex.FileTransfer.Server.Store.Postgres.Migrations
Simplex.FileTransfer.Server.Store.Postgres.Config
```
CPP guards (`#if defined(dbServerPostgres)`) in:
- `Store.hs` — Postgres `FromField`/`ToField` instances for XFTP-specific types if needed
- `Env.hs``XSCDatabase` constructor
- `Main.hs` — database CLI commands, store selection for `database` mode, Postgres imports
- `Server.hs` — Postgres-specific imports if needed
## Testing
- **Parameterized server tests**: Existing `xftpServerTests` refactored to accept a store type parameter (following SMP's `SpecWith (ASrvTransport, AStoreType)` pattern). The same server tests run against both STM and Postgres backends — STM tests run unconditionally, Postgres tests added under `#if defined(dbServerPostgres)` with `postgressBracket` for database lifecycle (drop → create → test → drop).
- **Unit tests**: `PostgresFileStore` operations — add/get/delete/block/expire, duplicate detection, auth errors
- **Migration round-trip**: STM store → export to StoreLog → import to Postgres → export back → verify StoreLog equality (including blocked file status)
- **Tests location**: in `tests/` alongside existing XFTP tests, guarded by `server_postgres` CPP flag
- **Test database**: PostgreSQL on `localhost:5432`, using a dedicated `xftp_server_test` schema (dropped and recreated per test run via `postgressBracket`, following SMP's test database lifecycle pattern)
- **Test fixtures**: `testXFTPStoreDBOpts :: DBOpts` with `createSchema = True`, `confirmMigrations = MCYesUp`, in `tests/XFTPClient.hs`
@@ -0,0 +1,648 @@
# XFTP PostgreSQL Backend — Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers-extended-cc:subagent-driven-development (if subagents available) or superpowers-extended-cc:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add PostgreSQL backend support to xftp-server as an alternative to STM + StoreLog, with bidirectional migration.
**Architecture:** Introduce `FileStoreClass` typeclass (IO-based, following `QueueStoreClass` pattern). Extract current STM store into `Store/STM.hs`, make `Server.hs` polymorphic, then add `Store/Postgres.hs` behind `server_postgres` CPP flag. `usedStorage` moves from store to `XFTPEnv` so the server manages quota tracking externally.
**Tech Stack:** Haskell, postgresql-simple, STM, fourmolu, cabal with CPP flags
**Design spec:** `plans/2026-03-25-xftp-postgres-backend-design.md`
---
## File Structure
**Existing files modified:**
- `src/Simplex/FileTransfer/Server/Store.hs` — rewritten: becomes typeclass + shared types
- `src/Simplex/FileTransfer/Server/Env.hs` — polymorphic `XFTPEnv s`, `XFTPStoreConfig` GADT
- `src/Simplex/FileTransfer/Server.hs` — polymorphic over `FileStoreClass s`
- `src/Simplex/FileTransfer/Server/StoreLog.hs` — update for IO store functions
- `src/Simplex/FileTransfer/Server/Main.hs` — INI config, dispatch, CLI commands
- `simplexmq.cabal` — new modules
- `tests/XFTPClient.hs` — Postgres test fixtures
- `tests/Test.hs` — Postgres test group
**New files created:**
- `src/Simplex/FileTransfer/Server/Store/STM.hs``STMFileStore` (extracted from current `Store.hs`)
- `src/Simplex/FileTransfer/Server/Store/Postgres.hs``PostgresFileStore` [CPP-guarded]
- `src/Simplex/FileTransfer/Server/Store/Postgres/Config.hs``PostgresFileStoreCfg` [CPP-guarded]
- `src/Simplex/FileTransfer/Server/Store/Postgres/Migrations.hs` — schema SQL [CPP-guarded]
- `tests/CoreTests/XFTPStoreTests.hs` — Postgres store unit tests [CPP-guarded]
---
## Task 1: Move `usedStorage` from `FileStore` to `XFTPEnv`
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Store.hs`
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- Modify: `src/Simplex/FileTransfer/Server.hs`
- [ ] **Step 1: Remove `usedStorage` from `FileStore` in `Store.hs`**
1. Remove `usedStorage :: TVar Int64` field from `FileStore` record (line 47).
2. Remove `usedStorage <- newTVarIO 0` from `newFileStore` (line 75) and drop the field from the record construction (line 76).
3. In `setFilePath` (line 92-97): remove `modifyTVar' (usedStorage st) (+ fromIntegral (size fileInfo))` — keep only `writeTVar filePath (Just fPath)`. Change pattern from `\FileRec {fileInfo, filePath}` to `\FileRec {filePath}` (fileInfo is now unused — `-Wunused-matches` error).
4. In `deleteFile` (line 112-119): remove `modifyTVar' usedStorage $ subtract (fromIntegral $ size fileInfo)`. Change outer pattern match from `FileStore {files, recipients, usedStorage}` to `FileStore {files, recipients}`. Change inner pattern from `Just FileRec {fileInfo, recipientIds}` to `Just FileRec {recipientIds}` (`fileInfo` is now unused — `-Wunused-matches` error).
5. In `blockFile` (line 122-127): remove `when deleted $ modifyTVar' usedStorage $ subtract (fromIntegral $ size fileInfo)`. Change pattern match from `st@FileStore {usedStorage}` to `st`. The `deleted` parameter and `fileInfo` in the inner pattern become unused — prefix with `_` or remove from pattern to avoid `-Wunused-matches`.
- [ ] **Step 2: Add `usedStorage` to `XFTPEnv` in `Env.hs`**
1. Add `usedStorage :: TVar Int64` field to `XFTPEnv` record (between `store` and `storeLog`, line 93).
2. In `newXFTPServerEnv` (line 112-126): replace lines 117-118:
```
used <- countUsedStorage <$> readTVarIO (files store)
atomically $ writeTVar (usedStorage store) used
```
with:
```
usedStorage <- newTVarIO =<< countUsedStorage <$> readTVarIO (files store)
```
3. Add `usedStorage` to the `pure XFTPEnv {..}` construction.
- [ ] **Step 3: Update all `usedStorage` access sites in `Server.hs`**
1. Line 552: `us <- asks $ usedStorage . store` → `us <- asks usedStorage`.
2. Line 569: `us <- asks $ usedStorage . store` → `us <- asks usedStorage`.
3. Line 639: `usedStart <- readTVarIO $ usedStorage st` → `usedStart <- readTVarIO =<< asks usedStorage`.
4. Line 647: `usedEnd <- readTVarIO $ usedStorage st` → `usedEnd <- readTVarIO =<< asks usedStorage`.
5. Line 694: `FileStore {files, usedStorage} <- asks store` → split into `FileStore {files} <- asks store` and `usedStorage <- asks usedStorage`.
6. In `deleteOrBlockServerFile_` (line 620): after `void $ atomically $ storeAction st`, add usedStorage adjustment — `us <- asks usedStorage` then `atomically $ modifyTVar' us $ subtract (fromIntegral $ size fileInfo)` when file had a path (check `path` from `readTVarIO filePath` earlier in the function).
- [ ] **Step 4: Build and verify**
Run: `cabal build`
- [ ] **Step 5: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 6: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs
git add src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs
git commit -m "refactor(xftp): move usedStorage from FileStore to XFTPEnv"
```
---
## Task 2: Add `getUsedStorage`, `getFileCount`, `expiredFiles` functions
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Store.hs`
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- Modify: `src/Simplex/FileTransfer/Server.hs`
- [ ] **Step 1: Add three new functions to `Store.hs`**
1. Add to exports: `getUsedStorage`, `getFileCount`, `expiredFiles`.
2. Remove `expiredFilePath` from exports AND delete the function definition (dead code → `-Wunused-binds` error). Also remove `($>>=)` from import `Simplex.Messaging.Util (ifM, ($>>=))` → `Simplex.Messaging.Util (ifM)` — `$>>=` was only used by `expiredFilePath`.
3. Add import: `qualified Data.Map.Strict as M` (needed for `M.foldl'` in `getUsedStorage` and `M.toList` in `expiredFiles`).
4. Implement:
```haskell
getUsedStorage :: FileStore -> IO Int64
getUsedStorage FileStore {files} =
M.foldl' (\acc FileRec {fileInfo = FileInfo {size}} -> acc + fromIntegral size) 0 <$> readTVarIO files
getFileCount :: FileStore -> IO Int
getFileCount FileStore {files} = M.size <$> readTVarIO files
expiredFiles :: FileStore -> Int64 -> Int -> IO [(SenderId, Maybe FilePath, Word32)]
expiredFiles FileStore {files} old _limit = do
fs <- readTVarIO files
fmap catMaybes . forM (M.toList fs) $ \(sId, FileRec {fileInfo = FileInfo {size}, filePath, createdAt = RoundedSystemTime createdAt}) ->
if createdAt + fileTimePrecision < old
then do
path <- readTVarIO filePath
pure $ Just (sId, path, size)
else pure Nothing
```
5. Add imports: `Data.Maybe (catMaybes)`, `Data.Word (Word32)` (note: `qualified Data.Map.Strict as M` already added in item 3).
- [ ] **Step 2: Replace `countUsedStorage` in `Env.hs`**
1. Replace `countUsedStorage <$> readTVarIO (files store)` with `getUsedStorage store` in `newXFTPServerEnv`.
2. Remove `countUsedStorage` function definition and its export.
3. Remove `qualified Data.Map.Strict as M` import if no longer used.
- [ ] **Step 3: Update `restoreServerStats` in `Server.hs` to use `getFileCount`**
In `restoreServerStats` (line 694-696): replace `FileStore {files} <- asks store` and `_filesCount <- M.size <$> readTVarIO files` with `st <- asks store` and `_filesCount <- liftIO $ getFileCount st` (eliminates the `FileStore` pattern match — `files` binding no longer needed).
- [ ] **Step 4: Replace `expireServerFiles` iteration in `Server.hs`**
1. Replace the body of `expireServerFiles` (lines 636-660). Remove `files' <- readTVarIO (files st)` and the `forM_ (M.keys files')` loop.
2. New body: call `expiredFiles st old 10000` in a loop. For each `(sId, filePath_, fileSize)` in returned list: apply `itemDelay`, remove disk file if present, call `atomically $ deleteFile st sId`, adjust `usedStorage` TVar by `fileSize`, increment `filesExpired` stat. Loop until `expiredFiles` returns `[]`.
3. Remove `Data.Map.Strict` import from Server.hs if no longer needed (was used for `M.size` and `M.keys` — now replaced by `getFileCount` and `expiredFiles`).
- [ ] **Step 5: Build and verify**
Run: `cabal build`
- [ ] **Step 6: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 7: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs
git add src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs
git commit -m "refactor(xftp): add getUsedStorage, getFileCount, expiredFiles store functions"
```
---
## Task 3: Change `Store.hs` functions from STM to IO
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Store.hs`
- Modify: `src/Simplex/FileTransfer/Server.hs`
- Modify: `src/Simplex/FileTransfer/Server/StoreLog.hs`
- [ ] **Step 1: Change all Store.hs function signatures from STM to IO**
For each of: `addFile`, `setFilePath`, `addRecipient`, `getFile`, `deleteFile`, `blockFile`, `deleteRecipient`, `ackFile`:
1. Change return type from `STM (Either XFTPErrorType ...)` to `IO (Either XFTPErrorType ...)` (or `STM ()` to `IO ()` for `deleteRecipient`).
2. Wrap the function body in `atomically $ do ...`.
3. Keep `withFile` and `newFileRec` as internal STM helpers (called inside the `atomically` blocks).
- [ ] **Step 2: Update Server.hs call sites — remove `atomically` wrappers**
1. Line 563 (`receiveServerFile`): change `atomically $ writeTVar filePath (Just fPath)` → add `st <- asks store` then `void $ liftIO $ setFilePath st senderId fPath` (design call site #1 — `store` is not in scope in `receiveServerFile`'s `receive` helper, so bind via `asks`; `void` avoids `-Wunused-do-bind` warning on the `Either` result).
2. Line 453 (`verifyXFTPTransmission`): split `atomically $ verify =<< getFile st party fId` into: `liftIO (getFile st party fId)` (IO→M lift), then pattern match on result, use `readTVarIO (fileStatus fr)` instead of `readTVar`.
3. Lines 371, 377 (control port `CPDelete`/`CPBlock`): change `ExceptT $ atomically $ getFile fs SFRecipient fileId` → `ExceptT $ liftIO $ getFile fs SFRecipient fileId` (inside `unliftIO u $ do` block which runs in M monad — `liftIO` required to lift IO into M).
4. Line 508 (`addFile` in `createFile`): the `ExceptT $ addFile st sId file ts EntityActive` — `addFile` is now IO, `ExceptT` wraps IO directly. Remove any `atomically`.
5. Line 514 (`addRecipient`): same — `ExceptT . addRecipient st sId` works directly in IO.
6. Line 516 (`retryAdd`): change parameter type from `(XFTPFileId -> STM (Either XFTPErrorType a))` to `(XFTPFileId -> IO (Either XFTPErrorType a))`. Line 520: change `atomically (add fId)` to `liftIO (add fId)`.
7. Line 605 (`ackFileReception`): change `atomically $ deleteRecipient st rId fr` to `liftIO $ deleteRecipient st rId fr`.
8. Line 620 (`deleteOrBlockServerFile_`): change third parameter type from `(FileStore -> STM (Either XFTPErrorType ()))` to `(FileStore -> IO (Either XFTPErrorType ()))`. Line 626: change `void $ atomically $ storeAction st` to `void $ liftIO $ storeAction st`.
9. `expireServerFiles` `delete` helper: change `atomically $ deleteFile st sId` to `liftIO $ deleteFile st sId` (deleteFile is now IO; `liftIO` required because the helper runs in M monad, not IO).
- [ ] **Step 3: Update `StoreLog.hs` — remove `atomically` from replay**
In `readFileStore` (line 93), function `addToStore`:
1. Change `atomically (addToStore lr)` to `addToStore lr` — store functions are now IO.
2. The `addToStore` body calls `addFile`, `setFilePath`, `deleteFile`, `blockFile`, `ackFile` — all IO now, no `atomically` needed.
3. For `AddRecipients`: `runExceptT $ mapM_ (ExceptT . addRecipient st sId) rcps` — `addRecipient` returns `IO (Either ...)`, so `ExceptT . addRecipient st sId` works directly.
- [ ] **Step 4: Build and verify**
Run: `cabal build`
- [ ] **Step 5: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 6: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/StoreLog.hs
git add src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/StoreLog.hs
git commit -m "refactor(xftp): change file store operations from STM to IO"
```
---
## Task 4: Extract `FileStoreClass` typeclass, move STM impl to `Store/STM.hs`
**Files:**
- Rewrite: `src/Simplex/FileTransfer/Server/Store.hs`
- Create: `src/Simplex/FileTransfer/Server/Store/STM.hs`
- Modify: `src/Simplex/FileTransfer/Server/StoreLog.hs`
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- Modify: `src/Simplex/FileTransfer/Server.hs`
- Modify: `simplexmq.cabal`
- [ ] **Step 1: Create `Store/STM.hs` — move all implementation code**
1. Create directory `src/Simplex/FileTransfer/Server/Store/`.
2. Create `src/Simplex/FileTransfer/Server/Store/STM.hs`.
3. Move from `Store.hs`: `FileStore` data type (rename to `STMFileStore`), all function implementations, internal helpers (`withFile`, `newFileRec`), all STM-specific imports.
4. Rename all `FileStore` references to `STMFileStore` in the new file.
5. Module declaration: `module Simplex.FileTransfer.Server.Store.STM` exporting only `STMFileStore (..)` — do NOT export standalone functions (`addFile`, `setFilePath`, etc.) to avoid name collisions with the typeclass methods from `Store.hs`.
- [ ] **Step 2: Rewrite `Store.hs` as the typeclass module**
1. Add `{-# LANGUAGE TypeFamilies #-}` pragma to `Store.hs` (required for `type FileStoreConfig s` associated type).
2. Keep in `Store.hs`: `FileRec (..)`, `FileRecipient (..)`, `RoundedFileTime`, `fileTimePrecision` definitions and their `StrEncoding` instance.
3. Add `FileStoreClass` typeclass:
```haskell
class FileStoreClass s where
type FileStoreConfig s
-- Lifecycle
newFileStore :: FileStoreConfig s -> IO s
closeFileStore :: s -> IO ()
-- File operations
addFile :: s -> SenderId -> FileInfo -> RoundedFileTime -> ServerEntityStatus -> IO (Either XFTPErrorType ())
setFilePath :: s -> SenderId -> FilePath -> IO (Either XFTPErrorType ())
addRecipient :: s -> SenderId -> FileRecipient -> IO (Either XFTPErrorType ())
getFile :: s -> SFileParty p -> XFTPFileId -> IO (Either XFTPErrorType (FileRec, C.APublicAuthKey))
deleteFile :: s -> SenderId -> IO (Either XFTPErrorType ())
blockFile :: s -> SenderId -> BlockingInfo -> Bool -> IO (Either XFTPErrorType ())
deleteRecipient :: s -> RecipientId -> FileRec -> IO ()
ackFile :: s -> RecipientId -> IO (Either XFTPErrorType ())
-- Expiration
expiredFiles :: s -> Int64 -> Int -> IO [(SenderId, Maybe FilePath, Word32)]
-- Stats
getUsedStorage :: s -> IO Int64
getFileCount :: s -> IO Int
```
4. Do NOT re-export from `Store/STM.hs` — this would create a circular module dependency (Store.hs imports Store/STM.hs, Store/STM.hs imports Store.hs). Consumers must import `Store.STM` directly where they need `STMFileStore`.
5. Remove all STM-specific imports that are no longer needed.
- [ ] **Step 3: Add `FileStoreClass` instance in `Store/STM.hs`**
1. Import `FileStoreClass` from `Simplex.FileTransfer.Server.Store`.
2. Inline all implementations directly in the instance body (do NOT delegate to standalone functions — the standalone names collide with typeclass method names, causing ambiguous occurrences for importers):
```haskell
instance FileStoreClass STMFileStore where
type FileStoreConfig STMFileStore = ()
newFileStore () = do
files <- TM.emptyIO
recipients <- TM.emptyIO
pure STMFileStore {files, recipients}
closeFileStore _ = pure ()
addFile st sId fileInfo createdAt status = atomically $ ...
setFilePath st sId fPath = atomically $ ...
-- ... (each method's body is the existing function body, inlined)
```
3. Remove the standalone top-level function definitions — they are now instance methods. Keep only `withFile` and `newFileRec` as internal helpers used by the instance methods.
- [ ] **Step 4: Update importers**
1. `Env.hs`: add `import Simplex.FileTransfer.Server.Store.STM (STMFileStore (..))`. Change `FileStore` → `STMFileStore` in `XFTPEnv` type and `newXFTPServerEnv`. Change `store <- newFileStore` to `store <- newFileStore ()` (typeclass method now takes `FileStoreConfig STMFileStore` which is `()`). Keep `import Simplex.FileTransfer.Server.Store` for `FileRec`, `FileRecipient`, `FileStoreClass`, etc.
2. `Server.hs`: add `import Simplex.FileTransfer.Server.Store.STM`. Change `FileStore` → `STMFileStore` in any explicit type annotations. Import `FileStoreClass` from `Simplex.FileTransfer.Server.Store`.
3. `StoreLog.hs`: add `import Simplex.FileTransfer.Server.Store.STM` to access concrete `STMFileStore` type and store functions used during log replay. Change `FileStore` → `STMFileStore` in `readWriteFileStore` and `writeFileStore` parameter types.
- [ ] **Step 5: Update cabal file**
Add `Simplex.FileTransfer.Server.Store.STM` to `exposed-modules` in the `!flag(client_library)` section, alongside existing XFTP server modules.
- [ ] **Step 6: Build and verify**
Run: `cabal build`
- [ ] **Step 7: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 8: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Store/STM.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/StoreLog.hs
git add src/Simplex/FileTransfer/Server/Store.hs src/Simplex/FileTransfer/Server/Store/STM.hs src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/StoreLog.hs simplexmq.cabal
git commit -m "refactor(xftp): extract FileStoreClass typeclass, move STM impl to Store.STM"
```
---
## Task 5: Make `XFTPEnv` and `Server.hs` polymorphic over `FileStoreClass`
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- Modify: `src/Simplex/FileTransfer/Server.hs`
- Modify: `src/Simplex/FileTransfer/Server/Main.hs`
- Modify: `tests/XFTPClient.hs` (if it calls `runXFTPServerBlocking` directly)
- [ ] **Step 1: Make `XFTPEnv` polymorphic in `Env.hs`**
1. Add `XFTPStoreConfig` GADT: `data XFTPStoreConfig s where XSCMemory :: Maybe FilePath -> XFTPStoreConfig STMFileStore`.
2. Change `data XFTPEnv` to `data XFTPEnv s` — field `store :: FileStore` becomes `store :: s`.
3. Change `newXFTPServerEnv :: XFTPServerConfig -> IO XFTPEnv` to `newXFTPServerEnv :: FileStoreClass s => XFTPStoreConfig s -> XFTPServerConfig -> IO (XFTPEnv s)`.
4. Pattern match on `XSCMemory storeLogPath` in `newXFTPServerEnv` body. Create store via `newFileStore ()`, storeLog via `mapM (`readWriteFileStore` st) storeLogPath`.
- [ ] **Step 2: Make `Server.hs` polymorphic**
1. Change `type M a = ReaderT XFTPEnv IO a` to `type M s a = ReaderT (XFTPEnv s) IO a`.
2. Add `FileStoreClass s =>` constraint to all functions using `M s a`. Use `forall s.` in signatures of functions that have `where`-block bindings with `M s` type annotations — `ScopedTypeVariables` requires explicit `forall` to bring `s` into scope for inner type signatures (matching SMP's `smpServer :: forall s. MsgStoreClass s => ...` pattern). Full list: `xftpServer`, `processRequest`, `verifyXFTPTransmission`, `processXFTPRequest` and all its `where`-bound functions (`createFile`, `addRecipients`, `receiveServerFile`, `sendServerFile`, `deleteServerFile`, `ackFileReception`, `retryAdd`, `addFileRetry`, `addRecipientRetry`), `deleteServerFile_`, `blockServerFile`, `deleteOrBlockServerFile_`, `expireServerFiles`, `randomId`, `getFileId`, `withFileLog`, `incFileStat`, `saveServerStats`, `restoreServerStats`, `randomDelay` (inside `#ifdef slow_servers` CPP block). Also update `encodeXftp` (line 236) and `runCPClient` (line 339) which use explicit `ReaderT XFTPEnv IO` instead of the `M` alias — change to `ReaderT (XFTPEnv s) IO`.
3. Change `runXFTPServerBlocking` and `runXFTPServer` to take `XFTPStoreConfig s` parameter.
4. Add `closeFileStore store` call to the server shutdown path (in the `finally` block or `stopServer` equivalent — after saving stats, before logging "Server stopped"). This ensures Postgres connection pool and `dbStoreLog` are properly closed. For STM this is a no-op.
- [ ] **Step 3: Update `Main.hs` dispatch**
1. In `runServer`: construct `XSCMemory (enableStoreLog $> storeLogFilePath)`.
2. Add dispatch function that calls the updated `runXFTPServer` (which creates `started` internally):
```haskell
run :: FileStoreClass s => XFTPStoreConfig s -> IO ()
run storeCfg = runXFTPServer storeCfg serverConfig
```
3. Call `run` with the `XSCMemory` config.
- [ ] **Step 4: Update test helper if needed**
If `tests/XFTPClient.hs` calls `runXFTPServerBlocking` directly, update the call to pass an `XSCMemory` config. Check the `withXFTPServer` / `serverBracket` helper.
- [ ] **Step 5: Build and verify**
Run: `cabal build && cabal build test:simplexmq-test`
- [ ] **Step 6: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 7: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/Main.hs
git add src/Simplex/FileTransfer/Server/Env.hs src/Simplex/FileTransfer/Server.hs src/Simplex/FileTransfer/Server/Main.hs tests/XFTPClient.hs simplexmq.cabal
git commit -m "refactor(xftp): make XFTPEnv and server polymorphic over FileStoreClass"
```
---
## Task 6: Add Postgres config, migrations, and store skeleton
**Files:**
- Create: `src/Simplex/FileTransfer/Server/Store/Postgres/Config.hs`
- Create: `src/Simplex/FileTransfer/Server/Store/Postgres/Migrations.hs`
- Create: `src/Simplex/FileTransfer/Server/Store/Postgres.hs`
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- Modify: `simplexmq.cabal`
- [ ] **Step 1: Create `Store/Postgres/Config.hs`**
```haskell
module Simplex.FileTransfer.Server.Store.Postgres.Config
( PostgresFileStoreCfg (..),
defaultXFTPDBOpts,
)
where
import Simplex.Messaging.Agent.Store.Postgres.Options (DBOpts (..))
import Simplex.Messaging.Agent.Store.Shared (MigrationConfirmation)
data PostgresFileStoreCfg = PostgresFileStoreCfg
{ dbOpts :: DBOpts,
dbStoreLogPath :: Maybe FilePath,
confirmMigrations :: MigrationConfirmation
}
defaultXFTPDBOpts :: DBOpts
defaultXFTPDBOpts =
DBOpts
{ connstr = "postgresql://xftp@/xftp_server_store",
schema = "xftp_server",
poolSize = 10,
createSchema = False
}
```
- [ ] **Step 2: Create `Store/Postgres/Migrations.hs`**
Full migration module with `xftpServerMigrations :: [Migration]` and `m20260325_initial` containing CREATE TABLE SQL for `files` and `recipients` tables plus indexes. Follow SMP's `QueueStore/Postgres/Migrations.hs` pattern exactly: tuple list → `sortOn name . map migration`.
- [ ] **Step 3: Create `Store/Postgres.hs` with stub instance**
1. Define `PostgresFileStore` with `dbStore :: DBStore` and `dbStoreLog :: Maybe (StoreLog 'WriteMode)`.
2. `instance FileStoreClass PostgresFileStore` with `error "not implemented"` for all methods except `newFileStore` (calls `createDBStore` + opens `dbStoreLog`) and `closeFileStore` (closes both). `type FileStoreConfig PostgresFileStore = PostgresFileStoreCfg`.
3. Add `withDB`, `handleDuplicate`, `assertUpdated`, `withLog` helpers.
- [ ] **Step 4: Add `XSCDatabase` GADT constructor in `Env.hs` (CPP-guarded)**
```haskell
#if defined(dbServerPostgres)
import Simplex.FileTransfer.Server.Store.Postgres (PostgresFileStore)
import Simplex.FileTransfer.Server.Store.Postgres.Config (PostgresFileStoreCfg)
#endif
data XFTPStoreConfig s where
XSCMemory :: Maybe FilePath -> XFTPStoreConfig STMFileStore
#if defined(dbServerPostgres)
XSCDatabase :: PostgresFileStoreCfg -> XFTPStoreConfig PostgresFileStore
#endif
```
- [ ] **Step 5: Update cabal**
Add to existing `if flag(server_postgres)` block:
```
Simplex.FileTransfer.Server.Store.Postgres
Simplex.FileTransfer.Server.Store.Postgres.Config
Simplex.FileTransfer.Server.Store.Postgres.Migrations
```
- [ ] **Step 6: Build both ways**
Run: `cabal build && cabal build -fserver_postgres`
- [ ] **Step 7: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store/Postgres.hs src/Simplex/FileTransfer/Server/Store/Postgres/Config.hs src/Simplex/FileTransfer/Server/Env.hs
git add src/Simplex/FileTransfer/Server/Store/Postgres.hs src/Simplex/FileTransfer/Server/Store/Postgres/Config.hs src/Simplex/FileTransfer/Server/Store/Postgres/Migrations.hs src/Simplex/FileTransfer/Server/Env.hs simplexmq.cabal
git commit -m "feat(xftp): add PostgreSQL store skeleton with schema migration"
```
---
## Task 7: Implement `PostgresFileStore` operations
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Store/Postgres.hs`
- [ ] **Step 1: Implement `addFile`**
`INSERT INTO files (sender_id, file_size, file_digest, sender_key, file_path, created_at, status) VALUES (?,?,?,?,NULL,?,?)`. Catch unique violation with `handleDuplicate` → `DUPLICATE_`. Call `withLog "addFile"` after.
- [ ] **Step 2: Implement `getFile`**
For `SFSender`: `SELECT ... FROM files WHERE sender_id = ?`. Construct `FileRec` with `newTVarIO` per TVar field. `recipientIds = S.empty`.
For `SFRecipient`: `SELECT f.*, r.recipient_key FROM recipients r JOIN files f ON r.sender_id = f.sender_id WHERE r.recipient_id = ?`.
- [ ] **Step 3: Implement `setFilePath`**
`UPDATE files SET file_path = ? WHERE sender_id = ? AND file_path IS NULL`. Use `assertUpdated`. Call `withLog "setFilePath"`.
- [ ] **Step 4: Implement `addRecipient`**
`INSERT INTO recipients (recipient_id, sender_id, recipient_key) VALUES (?,?,?)`. `handleDuplicate` → `DUPLICATE_`. Call `withLog "addRecipient"`.
- [ ] **Step 5: Implement `deleteFile`, `blockFile`**
`deleteFile`: `DELETE FROM files WHERE sender_id = ?` (CASCADE). `withLog "deleteFile"`.
`blockFile`: `UPDATE files SET status = ? WHERE sender_id = ?`. `assertUpdated`. `withLog "blockFile"`.
- [ ] **Step 6: Implement `deleteRecipient`, `ackFile`**
`deleteRecipient`: `DELETE FROM recipients WHERE recipient_id = ?`. `withLog "deleteRecipient"`.
`ackFile`: same + return `Left AUTH` if 0 rows.
- [ ] **Step 7: Implement `expiredFiles`, `getUsedStorage`, `getFileCount`**
`expiredFiles`: `SELECT sender_id, file_path, file_size FROM files WHERE created_at + ? < ? LIMIT ?`.
`getUsedStorage`: `SELECT COALESCE(SUM(file_size), 0) FROM files`.
`getFileCount`: `SELECT COUNT(*) FROM files`.
- [ ] **Step 8: Add `ToField`/`FromField` instances**
For `RoundedFileTime` (Int64 wrapper), `ServerEntityStatus` (Text via StrEncoding), `C.APublicAuthKey` (Binary via `encodePubKey`/`decodePubKey`). Check SMP's `QueueStore/Postgres.hs` for existing instances to import.
- [ ] **Step 9: Wrap mutation operations in `uninterruptibleMask_`**
Operations that combine a DB write with a TVar update (e.g., `getFile` constructs `FileRec` with `newTVarIO`) must be wrapped in `E.uninterruptibleMask_` to prevent async exceptions from leaving inconsistent state. Follow SMP's `addQueue_`, `deleteStoreQueue` pattern.
- [ ] **Step 10: Build**
Run: `cabal build -fserver_postgres`
- [ ] **Step 11: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Store/Postgres.hs
git add src/Simplex/FileTransfer/Server/Store/Postgres.hs
git commit -m "feat(xftp): implement PostgresFileStore operations"
```
---
## Task 8: Add INI config, Main.hs dispatch, startup validation
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Main.hs`
- Modify: `src/Simplex/FileTransfer/Server/Env.hs`
- [ ] **Step 1: Update `iniFileContent` in `Main.hs`**
Add to `[STORE_LOG]` section: `store_files: memory`, commented-out `db_connection`, `db_schema`, `db_pool_size`, `db_store_log` keys. Follow SMP's `optDisabled'` pattern for commented defaults.
- [ ] **Step 2: Add `StartOptions` and `--confirm-migrations` flag**
```haskell
data StartOptions = StartOptions
{ confirmMigrations :: MigrationConfirmation
}
```
Add to `Start` command parser with default `MCConsole`. Thread through to `runServer`.
- [ ] **Step 3: Add store_files INI parsing and CPP-guarded Postgres dispatch**
In `runServer`: read `store_files` from INI (`fromRight "memory" $ lookupValue "STORE_LOG" "store_files" ini`). Add `"database"` branch (CPP-guarded) that constructs `PostgresFileStoreCfg` using `iniDBOptions ini defaultXFTPDBOpts` and `enableDbStoreLog'` pattern. Non-postgres build: `exitError`.
- [ ] **Step 4: Add `XSCDatabase` branch in `newXFTPServerEnv` (`Env.hs`)**
CPP-guarded pattern match on `XSCDatabase dbCfg`: `newFileStore dbCfg`, `storeLog = Nothing`.
- [ ] **Step 5: Add startup config validation**
Add `checkFileStoreMode` (CPP-guarded) before `run`: validate conflicting storeLog file + database mode, missing schema, etc. per design doc.
- [ ] **Step 6: Build both ways**
Run: `cabal build && cabal build -fserver_postgres`
- [ ] **Step 7: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Main.hs src/Simplex/FileTransfer/Server/Env.hs
git add src/Simplex/FileTransfer/Server/Main.hs src/Simplex/FileTransfer/Server/Env.hs
git commit -m "feat(xftp): add PostgreSQL INI config, store dispatch, startup validation"
```
---
## Task 9: Add database import/export CLI commands
**Files:**
- Modify: `src/Simplex/FileTransfer/Server/Main.hs`
- [ ] **Step 1: Add `Database` CLI command (CPP-guarded)**
Add `Database StoreCmd DBOpts` constructor to `CliCommand`. Add `database` subcommand parser with `import`/`export` subcommands + `dbOptsP defaultXFTPDBOpts`.
- [ ] **Step 2: Implement `importFileStoreToDatabase`**
1. `confirmOrExit` with database details.
2. Create temporary `STMFileStore`, replay StoreLog via `readWriteFileStore`.
3. Create `PostgresFileStore` with `createSchema = True`, `confirmMigrations = MCYesUp`.
4. Batch-insert files using PostgreSQL COPY protocol. Progress every 10k.
5. Batch-insert recipients using COPY protocol.
6. Verify counts: `SELECT COUNT(*)` — warn on mismatch.
7. Rename StoreLog to `.bak`.
8. Report counts.
- [ ] **Step 3: Implement `exportDatabaseToStoreLog`**
1. `confirmOrExit`. Fail if output file exists.
2. Create `PostgresFileStore` from config.
3. Open StoreLog for writing.
4. Fold over file records: write `AddFile` (with status), `AddRecipients`, `PutFile` per file.
5. Close StoreLog, report counts.
- [ ] **Step 4: Build**
Run: `cabal build -fserver_postgres`
- [ ] **Step 5: Format and commit**
```bash
fourmolu -i src/Simplex/FileTransfer/Server/Main.hs
git add src/Simplex/FileTransfer/Server/Main.hs
git commit -m "feat(xftp): add database import/export CLI commands"
```
---
## Task 10: Add Postgres tests
**Files:**
- Modify: `tests/XFTPClient.hs`
- Modify: `tests/Test.hs`
- Create: `tests/CoreTests/XFTPStoreTests.hs`
- [ ] **Step 1: Add test fixtures in `tests/XFTPClient.hs`**
```haskell
testXFTPStoreDBOpts :: DBOpts
testXFTPStoreDBOpts =
DBOpts
{ connstr = "postgresql://test_xftp_server_user@/test_xftp_server_db",
schema = "xftp_server_test",
poolSize = 10,
createSchema = True
}
```
Add `testXFTPDBConnectInfo :: ConnectInfo` matching the connection string.
- [ ] **Step 2: Add Postgres server test group in `tests/Test.hs`**
CPP-guarded block that runs existing `xftpServerTests` with Postgres store config, wrapped in `postgressBracket testXFTPDBConnectInfo`. Parameterize `withXFTPServer` to accept store config if needed.
- [ ] **Step 3: Create `tests/CoreTests/XFTPStoreTests.hs` — unit tests**
Test `PostgresFileStore` operations directly:
- `addFile` + `getFile SFSender` round-trip.
- `addFile` duplicate → `DUPLICATE_`.
- `getFile` nonexistent → `AUTH`.
- `setFilePath` + verify `WHERE file_path IS NULL` guard.
- `addRecipient` + `getFile SFRecipient` round-trip.
- `deleteFile` cascades recipients.
- `blockFile` + verify status.
- `expiredFiles` batch semantics.
- `getUsedStorage`, `getFileCount` correctness.
- [ ] **Step 4: Add migration round-trip test**
Create `STMFileStore` with test data (files + recipients + blocked status) → export to StoreLog → import to Postgres → export back → compare StoreLog files byte-for-byte.
- [ ] **Step 5: Build and run tests**
```bash
cabal build -fserver_postgres test:simplexmq-test
cabal test --test-show-details=streaming --test-option=--match="/XFTP/" -fserver_postgres
```
- [ ] **Step 6: Format and commit**
```bash
fourmolu -i tests/CoreTests/XFTPStoreTests.hs tests/XFTPClient.hs
git add tests/CoreTests/XFTPStoreTests.hs tests/XFTPClient.hs tests/Test.hs
git commit -m "test(xftp): add PostgreSQL backend tests"
```
+126
View File
@@ -0,0 +1,126 @@
# BBS+ Bindings for simplexmq
Haskell FFI bindings to libbbs for BBS+ signatures. General-purpose - the module knows nothing about specific applications.
## How BBS+ works
BBS+ signs a fixed list of N messages. Each message is an arbitrary byte array. The signer signs all N messages at once with one signature.
The holder of the signature can then generate a proof that selectively discloses some messages and hides others. The verifier learns the disclosed messages and confirms they were signed by the signer, but learns nothing about the hidden messages. Different proofs from the same signature are unlinkable.
Key constraint: the total number of messages N is fixed at signing time. The verifier must know N. A proof generated from a 3-message signature cannot be verified as a 2-message proof.
## Types
```haskell
newtype BBSSecretKey = BBSSecretKey ByteString -- 32 bytes
newtype BBSPublicKey = BBSPublicKey ByteString -- 96 bytes (BLS12-381 G2 point)
newtype BBSSignature = BBSSignature ByteString -- 80 bytes
newtype BBSProof = BBSProof ByteString -- 272 + 32 * numUndisclosed bytes
newtype BBSHeader = BBSHeader ByteString -- always-disclosed context (e.g. protocol identifier)
newtype BBSPresHeader = BBSPresHeader ByteString -- random nonce for proof unlinkability
```
All newtypes get StrEncoding (base64url), ToJSON/FromJSON (via strToJSON/strParseJSON), Eq, Show.
## Functions
```haskell
bbsKeyGen :: IO (Either String BBSKeyPair) -- BBSKeyPair = (BBSPublicKey, BBSSecretKey)
-- pk is derived from sk internally, so it is not a parameter
bbsSign
:: BBSSecretKey
-> BBSHeader -- always-disclosed context
-> [ByteString] -- all N messages
-> IO (Either String BBSSignature)
-- C order: pk, signature, header, presentation_header, disclosed_indexes, messages
bbsProofGen
:: BBSPublicKey
-> BBSSignature
-> BBSHeader -- must match what was signed
-> BBSPresHeader -- random nonce bound into the proof
-> [Int] -- disclosed indexes (0-based)
-> [ByteString] -- all N messages (needed internally, hidden ones not revealed in proof)
-> IO (Either String BBSProof)
-- C order: pk, proof, header, presentation_header, disclosed_indexes, n, messages
bbsProofVerify
:: BBSPublicKey
-> BBSProof
-> BBSHeader -- must match what was signed
-> BBSPresHeader -- must match what was used in bbsProofGen
-> [Int] -- disclosed indexes
-> Int -- total message count N
-> [ByteString] -- disclosed messages only
-> IO Bool
```
## How applications use it
An application defines:
- A message layout: which index means what
- Which indexes are disclosed vs hidden
- How to encode application values as ByteString messages
### Badge example (in simplex-chat, not in this module)
Message layout (always 3 messages):
- Index 0: master secret (32 random bytes) - HIDDEN
- Index 1: expiry (UTF-8 encoded timestamp string) - DISCLOSED
- Index 2: badge type (UTF-8 encoded, e.g. "supporter") - DISCLOSED
Signing (v2, on the server):
```
bbsSign sk header [ms, encodeUtf8 "2026-07-31", encodeUtf8 "supporter"]
```
Proof generation (v2, on the client):
```
bbsProofGen pk sig header presHeader [1, 2] [ms, encodeUtf8 "2026-07-31", encodeUtf8 "supporter"]
```
Proof verification (v1, on the recipient):
```
bbsProofVerify pk proof header presHeader 3 [1, 2] [encodeUtf8 "2026-07-31", encodeUtf8 "supporter"]
```
The recipient only sees the proof, presentationHeader, expiry string, and badge type string. They verify these were signed by the server (pk is hardcoded). They never see the master secret.
Expiry is always present as a string. Monthly badges use a date like `"2026-07-31"`, lifetime badges use `"lifetime"`. BBS+ doesn't interpret the bytes - expiry semantics are the application's responsibility. This keeps the message count fixed at 3 for all badge types.
## libbbs C API mapping
```c
int bbs_keygen_full(ciphersuite, sk, pk)
int bbs_sign(ciphersuite, sk, pk, signature, header, header_len, n, messages, message_lens)
int bbs_proof_gen(ciphersuite, pk, signature, proof, header, header_len, presentation_header, presentation_header_len, disclosed_indexes, disclosed_indexes_len, n, messages, message_lens)
int bbs_proof_verify(ciphersuite, pk, proof, proof_len, header, header_len, presentation_header, presentation_header_len, disclosed_indexes, disclosed_indexes_len, n, messages, message_lens)
```
We use `bbs_sha256_ciphersuite`. The header parameter is exposed in all Haskell functions - the application decides what to put there. Tests use `"SimpleX"` as header.
The `presentation_header` parameter is what we call `presentationHeader`.
In `bbs_proof_verify`, the `n` parameter is the total number of messages (not the number of disclosed messages). The `messages` array contains only the disclosed messages, and `disclosed_indexes` maps each to its position in the original message list.
## Build
Submodules in cbits/:
- `cbits/libbbs` - https://github.com/Fraunhofer-AISEC/libbbs
- `cbits/blst` - https://github.com/supranational/blst (libbbs dependency)
C sources in cabal: `cbits/blst/src/server.c`, `cbits/blst/build/assembly.S`, libbbs source files.
Include dirs: `cbits/blst/bindings/`, `cbits/blst/src/`, `cbits/libbbs/include/`, `cbits/libbbs/src/`.
C flags: `-D__BLST_PORTABLE__` for cross-CPU-generation compatibility.
## Tests
- Keygen produces keys of correct size
- Sign + proofGen + proofVerify roundtrip succeeds
- Tampered proof fails verification
- Tampered disclosed message fails verification
- Wrong public key fails verification
- Two proofs from same credential with different nonces both verify
- Proof size matches expected (272 + 32 * numUndisclosed)
@@ -0,0 +1,57 @@
## Root cause: orphaned `Sub` entries in the service client's `subscriptions` map
**The leak is service-specific and was introduced by PR #1667 "messaging services" (`f0b7a4be`).** A long-lived messaging-service connection accumulates per-queue `Sub` records in its `Client.subscriptions` map that are **never removed** when the associated queues are deleted or unassociated — only the counter is decremented. Over normal queue churn the map grows monotonically for the entire lifetime of the service connection.
### The proof — an asymmetry between two handlers in `serverThread`
Both individual queue subscriptions and service subscriptions store a `Sub` per queue in `Client.subscriptions` (= `clientSubs` for the SMP subscriber thread, wired at `Server.hs:189`). When a queue ends/is deleted, the two paths diverge:
**Individual subscriber — entry IS removed** (`Server.hs:332`, `346`):
```haskell
CSAEndSub qId -> atomically (endSub c qId) >>= a unsub_ -- :332
...
endSub c qId = TM.lookupDelete qId (clientSubs c) >>= (removeWhenNoSubs c $>) -- :346
```
**Service subscriber — entry is NOT removed** (`Server.hs:336-340`):
```haskell
CSAEndServiceSub qId -> atomically $ do
modifyTVar' (clientServiceSubs c) decrease -- decrements serviceSubsCount
modifyTVar' totalServiceSubs decrease -- decrements global count
where decrease = subtractServiceSubs (1, queueIdHash qId)
-- never touches (clientSubs c) — the Sub for qId stays forever
```
### Where the orphaned entries are added (both new in this PR)
- `Server.hs:1860-1862` — on service subscribe (`SSUB`), one `Sub` inserted per queue that has a pending message.
- `Server.hs:2039-2043` (`newServiceDeliverySub`) — on **every** `SEND` to a service-associated queue with no existing sub, a `Sub` is inserted into the service client's `subscriptions`. After delivery the thread state resets to `NoSub` (`:2069`) but the map entry remains as a "already delivering" marker (`:1856-1859`).
### Why they leak
The only places the service client's `subscriptions` map is cleared are:
- `clientDisconnected``swapTVar subscriptions M.empty` (`Server.hs:1097`) — only on disconnect.
- `CSADecreaseSubs``swapTVar (clientSubs c) M.empty` (`Server.hs:343`) — only on full service takeover by another connection.
- `delQueueAndMsgs``TM.lookupDelete entId $ subscriptions clnt` (`Server.hs:2164`) — but `clnt` here is **the recipient deleting its own queue, not the service client**. The service's entry for that queue is reached only via the `CSDeleted → endServiceSub → CSAEndServiceSub` path (`Server.hs:306, 313, 336`), which decrements the counter but leaves the map entry.
**Concrete scenario (fully traced):** Service `S` subscribes (`SSUB`) and stays connected for days. Recipient `R` owns service-associated queue `Q`. A `SEND` to `Q` inserts a `Sub` into `S.subscriptions[Q]` (`:2042`). `R` later deletes `Q``delQueueAndMsgs` runs on `R`'s connection, removes `Q` from `R.subscriptions`, decrements counters, enqueues `CSDeleted Q (Just S)` (`:2167`) → `serverThread` runs `CSAEndServiceSub Q` for `S` (`:336`), decrementing `S.serviceSubsCount` but **leaving `S.subscriptions[Q]` in place**. Net: one orphaned `Sub` (record + 2 TVars) per service-associated queue ever deleted/unassociated, never reclaimed until `S` disconnects. The logical counter `serviceSubsCount` correctly drops, so the map size diverges from the counter — making the leak invisible to the existing service-sub metric.
### Verdict
This is a deterministic, static-provable memory leak — no production logging needed to confirm the existence; the asymmetry between `CSAEndSub` (removes) and `CSAEndServiceSub` (doesn't) is the smoking gun. It is specific to messaging-service certificate clients, which is exactly the population added by the services/certificate PR.
### Secondary findings (lower impact, same PR area, not the primary cause)
- **`forkClient` register-after-fork race** (`Server.hs:1356-1359`): if the forked action's `finally` delete (`:1358`) runs before the parent's `IM.insert` (`:1359`), a `Weak ThreadId` of a dead thread is left in `endThreads` until disconnect. Pre-existing, tiny per-entry, but exercised far more by the PR's higher END/DELD volume.
- **Wrong-client counter decrement** (`Server.hs:2166`): `delQueueAndMsgs` decrements `serviceSubsCount` of the *deleting* client, not the service; harmless for non-service deleters (floored at 0) but corrupts accounting if a service deletes its own queue.
---
### Recommended fix (mirror `endSub` in the service path)
Make `CSAEndServiceSub` also delete the per-queue `Sub` and cancel its delivery thread, exactly as `CSAEndSub`/`endSub` do for individual subscribers. Roughly:
```haskell
CSAEndServiceSub qId -> do
s_ <- atomically $ do
modifyTVar' (clientServiceSubs c) decrease
modifyTVar' totalServiceSubs decrease
TM.lookupDelete qId (clientSubs c) <* removeWhenNoSubs c
forM_ unsub_ $ \unsub -> mapM_ unsub s_
where decrease = subtractServiceSubs (1, queueIdHash qId)
```
@@ -0,0 +1,152 @@
# Server: batched SUB command processing
Implementation plan for Part 1 of [RFC 2026-03-28-subscription-performance](../rfcs/2026-03-28-subscription-performance.md).
## Current state
When a batch of ~135 SUB commands arrives, the server already batches:
- Queue record lookups (`getQueueRecs` in `receive`, Server.hs:1151)
- Command verification (`verifyLoadedQueue`, Server.hs:1152)
But command processing is per-command (`foldrM process` in `client`, Server.hs:1372-1375). Each SUB calls `subscribeQueueAndDeliver` which calls `tryPeekMsg` - one DB query per queue. For Postgres, that's ~135 individual `SELECT ... FROM messages WHERE recipient_id = ? ORDER BY message_id ASC LIMIT 1` queries per batch.
## Goal
Replace ~135 individual message peek queries with 1 batched query per batch. No protocol changes.
## Implementation
### Step 1: Add `tryPeekMsgs` to MsgStoreClass
File: `src/Simplex/Messaging/Server/MsgStore/Types.hs`
Add to `MsgStoreClass`:
```haskell
tryPeekMsgs :: s -> [StoreQueue s] -> ExceptT ErrorType IO (Map RecipientId Message)
```
Returns a map from recipient ID to earliest pending message for each queue that has one. Queues with no messages are absent from the map.
### Step 2: Parameterize `deliver` to accept pre-fetched message
File: `src/Simplex/Messaging/Server.hs`
Currently `deliver` (inside `subscribeQueueAndDeliver`, line 1641) calls `tryPeekMsg ms q`. Add a parameter for an optional pre-fetched message:
```haskell
deliver :: Maybe Message -> (Bool, Maybe Sub) -> M s ResponseAndMessage
deliver prefetchedMsg (hasSub, sub_) = do
stats <- asks serverStats
fmap (either ((,Nothing) . err) id) $ liftIO $ runExceptT $ do
msg_ <- maybe (tryPeekMsg ms q) (pure . Just) prefetchedMsg
...
```
When `Nothing` is passed, falls back to individual `tryPeekMsg` (existing behavior). When `Just msg` is passed, uses it directly (batched path).
### Step 3: Pre-fetch messages before the processing loop
File: `src/Simplex/Messaging/Server.hs`
Currently (lines 1372-1375):
```haskell
forever $
atomically (readTBQueue rcvQ)
>>= foldrM process ([], [])
>>= \(rs_, msgs) -> ...
```
Add a pre-fetch step before the existing loop:
```haskell
forever $ do
batch <- atomically (readTBQueue rcvQ)
msgMap <- prefetchMsgs batch
foldrM (process msgMap) ([], []) batch
>>= \(rs_, msgs) -> ...
```
`prefetchMsgs` scans the batch, collects queues from SUB commands that have a verified queue (`q_ = Just (q, _)`), calls `tryPeekMsgs` once, returns the map. For batches with no SUBs it returns an empty map (no DB call).
`process` passes the looked-up message (or Nothing) through to `processCommand` and down to `deliver`.
The `foldrM process` loop, `processCommand`, `subscribeQueueAndDeliver`, and all other command handlers stay structurally the same. Only `deliver` gains one parameter, and the `client` loop gains one pre-fetch call.
### Step 4: Review
Review the typeclass signature and server usage. Confirm the interface has the right shape before implementing store backends.
### Step 5: Implement for each store backend
#### Postgres
File: `src/Simplex/Messaging/Server/MsgStore/Postgres.hs`
Single query using `DISTINCT ON`:
```sql
SELECT DISTINCT ON (recipient_id)
recipient_id, msg_id, msg_ts, msg_quota, msg_ntf_flag, msg_body
FROM messages
WHERE recipient_id IN ?
ORDER BY recipient_id, message_id ASC
```
Build `Map RecipientId Message` from results.
#### STM
File: `src/Simplex/Messaging/Server/MsgStore/STM.hs`
Loop over queues, call `tryPeekMsg` for each, collect into map.
#### Journal
File: `src/Simplex/Messaging/Server/MsgStore/Journal.hs`
Loop over queues, call `tryPeekMsg` for each, collect into map.
### Step 6: Handle edge cases
1. **Mixed batches**: `prefetchMsgs` collects only SUB queues. Non-SUB commands get Nothing for the pre-fetched message and process unchanged.
2. **Already-subscribed queues**: Include in pre-fetch - `deliver` is called for re-SUBs too (delivers pending message).
3. **Service subscriptions**: The pre-fetch doesn't care about service state. `sharedSubscribeQueue` handles service association in STM; message peek is the same.
4. **Error queues**: Verification errors from `receive` are Left values in the batch. `prefetchMsgs` only looks at Right values with SUB commands.
5. **Empty pre-fetch**: If batch has no SUBs (e.g., all ACKs), `prefetchMsgs` returns empty map, no DB call made.
### Step 7: Batch other commands (future, not in scope)
The same pattern (pre-fetch before loop, parameterize handler) can extend to:
- `ACK` with `tryDelPeekMsg` - batch delete+peek
- `GET` with `tryPeekMsg` - same map lookup
Lower priority since these don't have the N-at-once pattern of subscriptions.
## File changes summary
| File | Change |
|---|---|
| `src/Simplex/Messaging/Server/MsgStore/Types.hs` | Add `tryPeekMsgs` to typeclass |
| `src/Simplex/Messaging/Server/MsgStore/Postgres.hs` | Implement `tryPeekMsgs` with batch SQL |
| `src/Simplex/Messaging/Server/MsgStore/STM.hs` | Implement `tryPeekMsgs` as loop |
| `src/Simplex/Messaging/Server/MsgStore/Journal.hs` | Implement `tryPeekMsgs` as loop |
| `src/Simplex/Messaging/Server.hs` | Add `prefetchMsgs`, parameterize `deliver` |
## Testing
1. Existing server tests must pass unchanged (correctness preserved).
2. Add a test that subscribes a batch of queues (some with pending messages, some without) and verifies all get correct SOK + MSG responses.
3. Prometheus metrics: existing `qSub` stat should still increment correctly.
## Performance expectation
For 300K queues across ~2200 batches:
- Before: ~300K individual DB queries
- After: ~2200 batched DB queries (one per batch of ~135)
- ~136x reduction in DB round-trips
@@ -0,0 +1,126 @@
# Server: batch queue service associations
When a batch of SUB or NSUB commands arrives from a service client, each command that needs a new or removed service association calls `setQueueService` individually - one DB write per command. For 135 commands per batch, that's 135 individual `UPDATE msg_queues` queries.
## Goal
Reduce to at most 2 DB queries per batch (one for rcv associations, one for ntf associations), using `UPDATE ... RETURNING recipient_id` to identify which queues were actually updated.
Also fuse message pre-fetch and association batching into a single batch preparation step with a clean contract.
## Contract
```haskell
prepareBatch :: Maybe ServiceId -> NonEmpty (VerifiedTransmission s) -> M s (Either ErrorType (Map RecipientId (Maybe Message, Maybe (Either ErrorType ()))))
```
`Left e` = batch-level failure (message pre-fetch or association query failed entirely). All SUBs/NSUBs in the batch get this error.
`Right map` = per-queue results as a tuple:
- `Maybe Message` - pre-fetched message for SUB queues, `Nothing` for NSUB or no message
- `Maybe (Either ErrorType ())` - association result. `Nothing` = no update needed. `Just (Right ())` = update succeeded. `Just (Left e)` = update failed for this queue.
One map, one lookup per queue. `processCommand` passes both values to `subscribeQueueAndDeliver` / `subscribeNotifications` -> `sharedSubscribeQueue`.
Queues not in the map (non-SUB/NSUB commands, failed verification) are not affected.
## prepareBatch implementation
One accumulating fold over the batch, collecting three lists:
- `subMsgQs :: [StoreQueue s]` - SUB queues for message pre-fetch
- `rcvAssocQs :: [StoreQueue s]` - SUB queues needing `rcv_service_id` update (`clntServiceId /= rcvServiceId qr`)
- `ntfAssocQs :: [StoreQueue s]` - NSUB queues needing `ntf_service_id` update (`clntServiceId /= ntfServiceId` from `NtfCreds`)
Classification reads from the already-loaded `QueueRec` in `VerifiedTransmission` - no extra DB query.
Then three store calls (each skipped if its list is empty):
1. `tryPeekMsgs ms subMsgQs` -> `Map RecipientId Message`
2. `setRcvQueueServices (queueStore ms) clntServiceId rcvAssocQs` -> `Set RecipientId`
3. `setNtfQueueServices (queueStore ms) clntServiceId ntfAssocQs` -> `Set RecipientId`
Then one pass to merge results into `Map RecipientId (Maybe Message, Maybe (Either ErrorType ()))`:
- For each SUB queue: `(M.lookup rId msgMap, assocResult rId rcvUpdated rcvAssocQs)`
- For each NSUB queue: `(Nothing, assocResult rId ntfUpdated ntfAssocQs)`
Where `assocResult rId updated assocQs` = if the queue was in `assocQs` (needed update), then `Just (Right ())` if `rId` is in `updated`, else `Just (Left AUTH)`. If not in `assocQs` (no update needed), `Nothing`.
If any of the three calls fails entirely, return `Left e`.
## Store interface
Replace the polymorphic `setQueueServices` with two plain functions in `QueueStoreClass`:
```haskell
setRcvQueueServices :: s -> Maybe ServiceId -> [q] -> IO (Set RecipientId)
setNtfQueueServices :: s -> Maybe ServiceId -> [q] -> IO (Set RecipientId)
```
No `SParty p` polymorphism. Each function knows its column.
### Postgres implementation
`setRcvQueueServices`:
```sql
UPDATE msg_queues SET rcv_service_id = ?
WHERE recipient_id IN ? AND deleted_at IS NULL
RETURNING recipient_id
```
`setNtfQueueServices`:
```sql
UPDATE msg_queues SET ntf_service_id = ?
WHERE recipient_id IN ? AND notifier_id IS NOT NULL AND deleted_at IS NULL
RETURNING recipient_id
```
After each batch query, for each queue in the returned set:
1. Read QueueRec TVar, update with new serviceId
2. Write store log entry
### STM implementation
Loop over queues, call existing per-item logic, collect succeeded `RecipientId`s into a Set.
## Downstream changes in Server.hs
### processCommand
Gains one parameter: `Map RecipientId (Maybe Message, Maybe (Either ErrorType ()))`.
SUB case: `M.lookup entId prepared` gives `Just (msg_, assocResult)` or `Nothing`. Pass both to `subscribeQueueAndDeliver`.
NSUB case: `M.lookup entId prepared` gives `Just (Nothing, assocResult)` or `Nothing`. Pass `assocResult` to `subscribeNotifications`.
Forwarded commands: pass `M.empty`.
### subscribeQueueAndDeliver
Takes `Maybe Message` and `Maybe (Either ErrorType ())` as before. No change in how it uses them.
### sharedSubscribeQueue
Takes `Maybe (Either ErrorType ())`. On paths needing association update:
- `Just (Left e)` -> return error
- `Just (Right ())` -> skip `setQueueService`, proceed with STM work
- `Nothing` -> no update needed, proceed with existing logic
## Implementation order (top-down)
1. Define the `prepareBatch` contract and thread one map through `processCommand` -> `subscribeQueueAndDeliver` / `subscribeNotifications` -> `sharedSubscribeQueue` (Server.hs)
2. Implement `prepareBatch` with the fold, three calls, and merge (Server.hs)
3. Add `setRcvQueueServices` and `setNtfQueueServices` to `QueueStoreClass` (Types.hs)
4. Implement for Postgres with batch `UPDATE ... RETURNING` (Postgres.hs)
5. Implement for STM as loop (STM.hs)
6. Implement for Journal as delegation (Journal.hs)
At step 2, store functions can initially be stubs returning empty sets. Steps 3-6 fill in the real implementations.
## Files changed
| File | Change |
|---|---|
| `src/Simplex/Messaging/Server.hs` | `prepareBatch` with fold + merge; one map parameter through `processCommand` -> `subscribeQueueAndDeliver` / `subscribeNotifications` -> `sharedSubscribeQueue` |
| `src/Simplex/Messaging/Server/QueueStore/Types.hs` | Add `setRcvQueueServices`, `setNtfQueueServices` to `QueueStoreClass` |
| `src/Simplex/Messaging/Server/QueueStore/Postgres.hs` | Implement with batch `UPDATE ... RETURNING` + per-item TVar/log updates |
| `src/Simplex/Messaging/Server/QueueStore/STM.hs` | Implement as loop |
| `src/Simplex/Messaging/Server/MsgStore/Journal.hs` | Delegate to underlying store |
+455
View File
@@ -0,0 +1,455 @@
# Server: SMP support for public namespaces
> **⚠ Implementation diverged from this plan.** Six audit rounds reshaped the
> original design. **The shipped code differs in several load-bearing ways:**
>
> - **Wire format**: `NameRecord` is now JSON (aeson), not the custom binary
> ABNF this plan documents. See `protocol/simplex-messaging.md` §Resolver
> commands and `src/Simplex/Messaging/Protocol.hs` ToJSON/FromJSON instances.
> - **No cache**: the TTL + FIFO + byte-cap cache, in-flight coalescing,
> `psqueues` dep, and `cache_*` INI keys are all gone. Every RSLV becomes
> one `eth_call` bounded by `rpcMaxConcurrency` + `rpcTimeoutMs`. See
> `src/Simplex/Messaging/Server/Names.hs`.
> - **No `allow_dangerous_colocation` flag**: the proxy co-location guard
> was demoted to a startup `logWarn` (the flag was always-on because
> `[PROXY]` has no enable toggle).
> - **Module shape**: `Names/Resolver.hs` was merged into `Names.hs`; only
> `Names/Eth/RPC.hs` and `Names/Eth/SNRC.hs` remain as separate modules.
> - **Test list**: of the 15 specs listed below, ~7 shipped; the rest were
> either superseded by the cache removal (CacheSpec) or deferred
> (ForwardedRslvSpec, MockRpcSpec, StartupGuardSpec, UrlValidationSpec,
> EipChecksumSpec).
>
> Sources of truth: `CHANGELOG.md` (release notes),
> `protocol/simplex-messaging.md` §Resolver commands (wire format),
> `src/Simplex/Messaging/Server/Names*.hs` (implementation). This file is
> retained as historical context; do not treat it as a specification.
Implementation plan for Part 2 of [RFC 2026-05-21-public-namespaces](https://github.com/simplex-chat/simplex-chat/blob/ep/namespace/docs/rfcs/2026-05-21-public-namespaces.md). Adds a forwarded-only `RSLV <lookup_key>` SMP command that returns `NAME <NameRecord>` read from the SNRC contract via a Reth+Nimbus JSON-RPC endpoint. Smp-server becomes name-capable by `[NAMES] enable: on`.
Out of scope: `Simplex.Messaging.Client` API, agent-side resolution flow, `ServerRoles.names` in the agent, default-router list, reverse resolution, multicoin/text records, state proofs.
## Architecture
```mermaid
sequenceDiagram
participant C as Client
participant P as Proxy (storage role)
participant N as Name server (names role)
participant E as Ethereum endpoint<br/>(Reth+Nimbus)
C ->> P: PFWD(enc(RSLV key))
P ->> N: RFWD(enc(RSLV key))
note over N: verifyTransmission True →<br/>vc SResolver (RSLV _) → VRVerified
N ->> N: cache lookup
alt cache miss
N ->> E: eth_call(SNRC, namehash(key))
E -->> N: ABI bytes
note over N: ABI decode + zero-owner check + cache insert
end
N -->> P: RFWD(enc(NAME rec | ERR AUTH))
P -->> C: PRES(enc(NAME rec | ERR AUTH))
```
RSLV is **forwarded-only** — direct RSLV is rejected `CMD PROHIBITED`. This preserves the RFC's two-server resolution: the name server sees the lookup key but never the client's IP, session, or identity.
## Protocol
Shared library: `src/Simplex/Messaging/Protocol.hs` and `src/Simplex/Messaging/Transport.hs`.
**Version.** `Transport.hs:226`: `namesSMPVersion = VersionSMP 20`. Bump `currentClientSMPRelayVersion`, `currentServerSMPRelayVersion`, `proxiedSMPRelayVersion` to 20. Pre-v20 binaries lack the `RSLV_` tag; v20 binaries with sessions negotiated at v < 20 reject `RSLV_` at the parameter parser. The proxied-version bump 18 → 20 is safe (v19's `RecipientService`/`NotifierService` aren't in the forwarded whitelist; v18's `BLOCKED info` is already version-branched at `Protocol.hs:1943`).
**Party kind.** Append `Resolver` to `Party` (line 335); add `SResolver` (line 349), `TestEquality` clause (line 361), `PartyI Resolver` (line 394). `queueParty SResolver = Nothing` (falls through line 412). `partyClientRole SResolver = Nothing`.
**`RSLV` command.**
```haskell
RSLV :: LookupKey -> Command Resolver
newtype LookupKey = LookupKey ByteString
instance Encoding LookupKey where
smpEncode (LookupKey s) = smpEncode s
smpP = do
n <- lenP
when (n > 64) $ fail "LookupKey too large"
LookupKey <$> A.take n
```
Name-syntax validation is client-side per RFC; the server treats the key as opaque bytes. Tag `"RSLV"`, version guard inside `protocolP v (CT SResolver RSLV_)`: `| v >= namesSMPVersion -> Cmd SResolver . RSLV <$> _smpP`.
**Testnet/mainnet selector**: how the `#testnet:name` namespace appears in `LookupKey` bytes is determined by the SNRC contract (Part 1) — confirm with Part 1 before merging.
**`NAME` response.**
```haskell
NAME :: NameRecord -> BrokerMsg
```
Tag `"NAME"`. Symmetric version guards on encode (in `encodeProtocol v`) and decode (in `protocolP v NAME_`): `| v >= namesSMPVersion -> ...`. `NameRecord` has **no `Encoding` typeclass instance** — the typeclass cannot version-branch. Use top-level helpers `nameRecBytes :: VersionSMP -> NameRecord -> ByteString` and `parseNameRec :: VersionSMP -> Parser NameRecord`, mirroring the `IDS QIK` precedent at `Protocol.hs:19121979`.
**`NameRecord` schema and wire layout.**
```haskell
data NameRecord = NameRecord
{ nrDisplayName :: Text -- ≤255 bytes UTF-8
, nrOwner :: NameOwner -- 20 raw bytes
, nrChannelLinks :: [NameLink]
, nrContactLinks :: [NameLink]
, nrAdminAddress :: Maybe Text
, nrAdminEmail :: Maybe Text
, nrExpiry :: Int64 -- Unix seconds, ≥ 0
, nrIsTest :: Bool
}
newtype NameOwner = NameOwner ByteString -- bare ctor NOT exported; smart ctor enforces length 20
newtype NameLink = NameLink Text -- bare ctor NOT exported; smart ctor enforces ≤1024 bytes
unNameOwner :: NameOwner -> ByteString
unNameOwner (NameOwner bs) = bs
unNameLink :: NameLink -> Text
unNameLink (NameLink t) = t
```
Field additions are gated by future SMP version bumps (matching the `IDS QIK` precedent at `Protocol.hs:19121979`) — no separate record-version field.
| Field | Encoding | Max bytes |
|---|---|---|
| `nrDisplayName` | 1-byte length prefix + UTF-8 | 1 + 255 |
| `nrOwner` | 20 raw bytes, no prefix | 20 |
| `nrChannelLinks`, `nrContactLinks` | 1-byte count + per-element (Word16 BE len + UTF-8); combined cap **8 entries** across both lists | 1 + Σ(2 + ≤1024) |
| `nrAdminAddress`, `nrAdminEmail` | `'0'` or `'1'` + (1-byte length + UTF-8 if `'1'`) | 1 + 1 + 255 |
| `nrExpiry` | two big-endian `Word32` | 8 |
| `nrIsTest` | `'T'` or `'F'` | 1 |
`Encoding NameLink` reads the Word16 length **before** `A.take` allocates — going through the existing `Large` wrapper allows up to 65 535 bytes per element. There is no `Encoding [a]` instance — use `smpEncodeList` / `smpListP` / a bounded variant:
```haskell
smpListPUpTo :: Encoding a => Int -> Parser [a]
smpListPUpTo cap = do
n <- lenP
when (n > cap) $ fail "list too long"
A.count n smpP
parseNameRec _v = do
nrDisplayName <- smpP
nrOwner <- smpP
nrChannelLinks <- smpListPUpTo 8
nrContactLinks <- smpListPUpTo (8 - length nrChannelLinks)
nrAdminAddress <- smpP
nrAdminEmail <- smpP
nrExpiry <- smpP
when (nrExpiry < 0) $ fail "expiry must be non-negative"
nrIsTest <- smpP
pure NameRecord{..}
```
Both list parsers fail at the count step before allocating; the second inherits the residual budget. Canonical encoding by construction: every primitive has exactly one valid byte form — two name servers reading the same SNRC state produce byte-identical responses.
**Wire-size budget.** `paddedProxiedTLength = 16226` is the plaintext input to `cbEncrypt` (`Server.hs:2117`); `pad` reserves 2 bytes → framed transmission ≤ 16 224 bytes. Combined-link cap 8 yields max payload ≈ 9 050 bytes — generous margin.
**Error semantics.** A single wire code: `ERR AUTH`. Per RFC, this collapses every failure (name not found, malformed key, names disabled, RPC unreachable, decode error, timeout). Resolver internally distinguishes the cause for stats only.
**Forwarded-only access.** Direct RSLV is rejected with `CMD PROHIBITED`. The shape of `THAuthServer` alone cannot discriminate direct from forwarded (`Transport.hs:852` sets `sessSecret' = Just _` for every v6+ direct client too). An explicit `forwarded :: Bool` flag is threaded through `verifyTransmission` (see below).
## Server changes
All edits in `src/Simplex/Messaging/Server.hs`.
**`forwarded :: Bool` plumbing.** Three signatures change:
- `verifyTransmission :: Bool -> ...` (line 1233) — direct path passes `False` (lines 11521153), forwarded path passes `True` (line 2129).
- `verifyLoadedQueue :: Bool -> ...` (line 1238) — receives the flag from `verifyTransmission` (lines 1235, 1240).
- `verifyQueueTransmission :: Bool -> ...` (line 1244) — receives and uses the flag.
New `vc` clauses inside `verifyQueueTransmission`:
```haskell
vc SResolver (RSLV _) | forwarded = VRVerified Nothing
| otherwise = VRFailed (CMD PROHIBITED)
vc SResolver _ = VRFailed (CMD PROHIBITED) -- defensive catch-all
```
**Forwarded whitelist** (`Server.hs:2132`):
```haskell
Cmd SResolver (RSLV _) -> True
```
**`processCommand` branch** (alongside line 1481):
```haskell
Cmd SResolver (RSLV (LookupKey key)) -> do
st <- asks (rslvStats . serverStats)
incStat (rslvReqs st)
asks namesEnv >>= \case
Nothing -> incStat (rslvDisabled st) $> response (corrId, NoEntity, ERR AUTH)
Just nenv -> liftIO (resolveName nenv key) >>= \case
Right rec -> incStat (rslvSucc st) $> response (corrId, NoEntity, NAME rec)
Left NotFound -> incStat (rslvNotFound st) $> response (corrId, NoEntity, ERR AUTH)
Left _ -> incStat (rslvEthErrs st) $> response (corrId, NoEntity, ERR AUTH)
```
**Shutdown.** Add `closeNamesEnv :: NamesEnv -> IO ()` calling `closeManager`. Wire into `closeServer` (`Server.hs:247`):
```haskell
closeServer = do
asks (smpAgent . proxyAgent) >>= liftIO . closeSMPClientAgent
asks namesEnv >>= liftIO . mapM_ closeNamesEnv
```
In-flight `resolveName` calls during shutdown receive `ConnectionClosed``EthHttpErr` → masked-leader cleanup runs → waiters unblock with `ERR AUTH`.
**`incStat` relocation.** Defined at `Server.hs:2220`, currently unexported. Move to `Server/Stats.hs` (one-line transplant + export) so `Resolver.hs` can use it.
**Co-located proxy warning.** `newEnv` logs a startup warning whenever `allowSMPProxy = True` and `namesConfig = Just _`. RSLV is the first slow forwarded command; on a proxy host it can serialise other forwarded commands on the same proxy-relay session up to `rpcTimeoutMs` per cache miss. The warning is not a hard refusal because `[PROXY]` has no `enable: on/off` toggle — proxy is always on for every smp-server. `forkForwardedCmd` async dispatch is the longer-term fix, tracked as a follow-up; once the proxy role is gateable per-server, the warning can be tightened back to a refusal.
## Resolver subtree
New module tree at `src/Simplex/Messaging/Server/Names/`:
| Module | Contents |
|---|---|
| `Names.hs` | Façade — re-exports `NamesConfig`, `NamesEnv`, `ResolveError`, `resolveName`, `newNamesEnv`, `closeNamesEnv`. |
| `Names/Resolver.hs` | All types + cache + in-flight + `resolveName`. Helpers exported directly (no `.Internal` per codebase convention). **Test seam**: `NamesEnv` holds `ethCall` as a function value, so tests construct stubs via `newNamesEnvWith`. |
| `Names/Eth/RPC.hs` | `EthRpcEnv`; `ethCallReal` via `http-client` + `withResponse` + `brReadSome rpcMaxResponseBytes`. JSON-RPC error / HTTP error split. `rpcMaxConcurrency` semaphore. `Authorization` header from `rpcAuth`. |
| `Names/Eth/SNRC.hs` | `EthAddress`, Keccak-256 namehash via `crypton`'s `Crypto.Hash.Algorithms.Keccak_256` (mirroring `Crypto.hs:10231025` for SHA3), hand-rolled bounded Solidity ABI codec, `getRecord` with zero-owner detection. **Ethereum's Keccak ≠ NIST SHA3-256.** |
**ABI codec invariants**, enforced before any allocation: `offset + 32 ≤ buf.length`; `offset + 32 + length ≤ buf.length`; `offset ≥ headEnd` (no backward jumps); every length ≤ per-field cap; `string[]` outer length × 32 ≤ buf.length; recursion depth ≤ 2; `uint256 → Int64` rejects if any high 24 bytes non-zero; UTF-8 via `decodeUtf8'` returns `EthDecodeErr`.
**Zero-owner → `NotFound`**: ENS-style resolvers return zeroed records for non-existent names. After ABI decode, if `nrOwner == NameOwner (B.replicate 20 0)` return `Left NotFound`.
**Errors.**
```haskell
data ResolveError = NotFound | EthHttpErr | EthRpcErr { rpcCode :: Int, rpcMessage :: Text }
| EthDecodeErr | TimedOut
```
All collapse to `ERR AUTH`. `EthRpcErr` carries JSON-RPC `error` object — method-not-found (SNRC not deployed at `snrc_address`) is logged immediately on the first error after a recent success: `logError "NAMES: JSON-RPC error from endpoint — check snrc_address: <code> <message>"`. No automatic retry.
**Cache.** TTL + FIFO eviction. `TVar (OrdPSQ LookupKey Word64 NameRecord, Int)` — priority = monotonic-ns at insert; the `Int` is running byte count. `cacheLookup` is one STM transaction (read, expiry-check, expired-delete-with-byte-decrement). `cacheInsert` is one STM transaction: while `size > cacheMaxEntries` OR `bytes + sizeOf(rec) > cacheMaxBytes`, `minView` to drop oldest, then `insert`. Byte counter prevents `100 000 × 9 KB ≈ 900 MB` worst-case blow-up.
**Request coalescing** (async-exception safe via `E.mask`):
```haskell
resolveName env bs = do
let k = LookupKey bs
now <- getMonotonicTimeNSec
atomically (cacheLookup env k now) >>= \case
Just rec -> incStat (rslvCacheHits ...) $> Right rec
Nothing -> do
incStat (rslvCacheMiss ...)
ticket <- atomically $ TM.lookup k (inflight env) >>= \case
Just mv -> pure (Waiter mv)
Nothing -> newEmptyTMVar >>= \mv -> TM.insert k mv (inflight env) $> Leader mv
case ticket of
Waiter mv -> atomically (readTMVar mv)
Leader mv -> E.mask $ \restore -> do
r <- restore (fetchOnceTimed env bs)
`E.catch` \(e :: E.SomeException) -> pure (Left (mapEthErr e))
atomically $ putTMVar mv r >> TM.delete k (inflight env)
case r of Right rec -> atomically (cacheInsert env k now rec); Left _ -> pure ()
pure r
fetchOnceTimed env bs =
System.Timeout.timeout (rpcTimeoutMs (config env) * 1000) (fetchOnce env bs) >>= \case
Just r -> pure r
Nothing -> pure (Left TimedOut)
```
`E.mask` ensures `putTMVar + TM.delete` runs even on async exception; `fetchOnceTimed` runs under `restore` so it remains interruptible. Waiters always see a value; the in-flight TMap entry is always removed.
`fetchOnce`, `mapEthErr`, `scrubUrl`, `cacheLookup`, `cacheInsert` are internal to `Resolver.hs`. `getMonotonicTimeNSec` from `GHC.Clock` — first monotonic-clock use in the codebase; clock-jump safe.
**STM contention.** Cache hits are read-only `readTVar` — STM scales. Cache writes under sustained miss traffic can retry; `CacheSpec` asserts < 5% retry at 4 readers + 1 writer @ 1k RPS. If observed higher, swap `TVar` for `IORef` + `atomicModifyIORef'`.
**Multicoin and text records** are not in `NameRecord`. If Part 1 contract returns them from `getRecord`, extend `NameRecord` and the wire-size budget. **Confirm with Part 1 author before implementing `Eth/SNRC.hs`.**
## Configuration
`ServerConfig` (`Env/STM.hs:142`) gains one field `namesConfig :: Maybe NamesConfig`. `Env` (`Env/STM.hs:261`) gains `namesEnv :: Maybe NamesEnv`. `newEnv` constructs it after `proxyAgent` (line 605) with the co-location guard.
```haskell
data NamesConfig = NamesConfig
{ ethereumEndpoint :: Text -- http(s), no userinfo, explicit port required
, snrcAddress :: NameOwner -- 20 bytes
, rpcAuth :: Maybe RpcAuth -- required when https & non-loopback host
, cacheSeconds :: Int -- 300
, cacheMaxEntries :: Int -- 100000
, cacheMaxBytes :: Int -- 67108864 (64 MB)
, rpcTimeoutMs :: Int -- 3000
, rpcMaxResponseBytes :: Int -- 262144 (256 KB)
, rpcMaxConcurrency :: Int -- 8
}
data RpcAuth = AuthBearer Text | AuthBasic Text Text
```
INI parsing in `Server/Main.hs`:
- `validateUrl` (using new `network-uri` dep): accepts only http(s), non-empty host, **explicit port** (rejects `http://localhost` defaulting to 80 while Reth is on 8545), no userinfo, no query/fragment. Rejects `https://...` without `rpc_auth` when host is non-loopback. On rejection: `logError` + `exitFailure`.
- `parseEthAddr`: accepts `0x[0-9a-fA-F]{40}` and the same without `0x`. Mixed-case → verify EIP-55 checksum and reject mismatch (catches typos).
- `parseRpcAuth`: reads optional `rpc_auth` key; format `bearer <token>` or `basic <user>:<pass>`.
- `scrubUrl`: strips userinfo from all log lines mentioning the endpoint, including inside `mapEthErr`.
- Transition-aware error logging: log immediately on first error after a recent success, then at most hourly while persisting + summary at every stats reset.
Default INI template (`Server/Main/Init.hs`, after `[PROXY]`):
```
[NAMES]
# Public-namespace resolution (SNRC on Ethereum).
# Requires an Ethereum JSON-RPC endpoint (Reth+Nimbus). See deployment guide.
# Cannot be combined with [PROXY] enable: on by default — see allow_dangerous_colocation.
# Restart required to change settings.
enable: off
# Same-host:
# ethereum_endpoint: http://127.0.0.1:8545
# Central Reth via Caddy:
# ethereum_endpoint: https://eth.simplex.chat:443
# rpc_auth: basic <username>:<password>
# snrc_address: 0x0000000000000000000000000000000000000000
# cache_seconds: 300
# cache_max_entries: 100000
# cache_max_bytes: 67108864
# rpc_timeout_ms: 3000
# rpc_max_response_bytes: 262144
# rpc_max_concurrency: 8
# allow_dangerous_colocation: off
```
Upgrade from a pre-v6.6 INI: missing `[NAMES]` section → disabled. No operator action required.
## Operator deployment
Two supported topologies. smp-server is agnostic — only `ethereum_endpoint` changes.
**Topology A (same-host)**: smp-server, Caddy (optional), Reth, Nimbus all on one box. `ethereum_endpoint: http://127.0.0.1:8545`.
**Topology B (central Reth, N smp-server hosts — recommended for fleets)**: one operator runs one eth host with Reth+Nimbus behind Caddy on public HTTPS. Each smp-server has its own credential.
```mermaid
flowchart LR
subgraph eth-host
Caddy["Caddy<br/>(public :443, basic auth)"]
Reth["Reth<br/>(127.0.0.1:8545)"]
Nimbus["Nimbus"]
Caddy --> Reth
Nimbus -- Engine API (jwt.hex) --> Reth
end
subgraph smp-host-1
S1["smp-server #1"]
end
subgraph smp-host-N
SN["smp-server #N"]
end
S1 -- HTTPS + Authorization --> Caddy
SN -- HTTPS + Authorization --> Caddy
Reth <-- Ethereum p2p --> internet
Nimbus <-- beacon sync --> internet
```
Sharing one Reth across **multiple operators** is **not** supported — collapses the RFC's two-server resolution privacy.
**Reth + Nimbus**: Reth (execution layer) holds Ethereum state on ~260 GB pruned NVMe; Nimbus (consensus light client) follows beacon-chain headers. Paired via Engine API on `127.0.0.1:8551` with a shared `jwt.hex`. Recommended Reth flags:
```bash
reth node \
--http.addr 127.0.0.1 \
--http.api eth \ # only eth namespace
--rpc.gascap 50000000 \ # cap gas per eth_call
--rpc.max-response-size 5242880 \ # 5 MB
--http.corsdomain none \
--authrpc.jwtsecret /opt/eth/jwt.hex \
--authrpc.addr 127.0.0.1 --authrpc.port 8551
```
**Caddy + Let's Encrypt + Basic auth** (Topology B):
```caddy
eth.simplex.chat {
basicauth {
smp-server-1 $2a$14$<bcrypt-hash-1>
smp-server-2 $2a$14$<bcrypt-hash-2>
}
log { format filter { wrap json; fields { request>headers>Authorization delete } } }
reverse_proxy 127.0.0.1:8545
}
```
Caddy auto-fetches Let's Encrypt cert. Each smp-server has its own credential; revoking one = delete the line. `Authorization` stripped from access logs. Port 80 needed for the ACME HTTP-01 challenge (use TLS-ALPN-01 or DNS-01 to drop it). The threat being defended against is DoS (SNRC state is public); mTLS would be overkill. WireGuard/Tailscale are alternative network-layer approaches — both compatible with the plan.
**Capacity.** One Reth+Nimbus box handles a realistic operator fleet by 101000× margin. Per-smp-server peak RSLV ≈ 1700 RPS (pessimistic); cache hit rate ≥ 95% → ~85 RPS cache miss per smp-server; 10 smp-servers → ~850 RPS aggregate cache miss reaching Reth; Reth `eth_call` throughput on warm NVMe ≈ 1k10k RPS. Sizing: 8 vCPU, 32 GB RAM, 1 TB NVMe is comfortable. Scale-out path: more Reth+Nimbus pairs, smp-servers round-robin or shard.
## Implementation
**Order**:
1. Protocol: party/SParty/PartyI, RSLV+tag, NAME+tag, NameRecord + helpers, version constants in `Transport.hs`.
2. `verifyTransmission`/`verifyLoadedQueue`/`verifyQueueTransmission` `forwarded :: Bool` flag + `vc SResolver` clauses.
3. Forwarded whitelist + `processCommand` branch + `incStat` move to `Stats.hs`.
4. Env plumbing: `Server/Env/STM.hs`, `Server/Main.hs` INI parse, `Server/Main/Init.hs` template.
5. Resolver subtree: `Eth/SNRC.hs``Eth/RPC.hs``Resolver.hs`.
6. `NameResolverStats` sub-record + CSV log + Prometheus `names =` block.
7. Replace stub in (3) with real `resolveName`.
8. Tests.
9. `protocol/simplex-messaging.md`: header version line 1 (`19 → 20`), sentence at line 86, version-history list (lines 93105) v20 entry, TOC (lines 2568) "Resolver commands" subsection, new section with ABNF + byte layout + error semantics, "Router security requirements" paragraph about names-role outbound HTTP, cross-ref `Transport.hs:226`.
10. `CHANGELOG.md`: v6.6 entry.
**Cabal** (`simplexmq.cabal`): bump `version: 6.6.0.0`. Add to `if !flag(client_library)` block: `http-client >=0.7 && <0.8`, `http-client-tls >=0.3 && <0.4`, `network-uri >=2.6 && <2.7`, `psqueues >=0.2.7 && <0.3`. Expose 4 new `Server.Names.*` modules in the same block. `crypton` already provides `Keccak_256`.
**Files changed**:
| File | Change |
|---|---|
| `Protocol.hs` | Resolver party + RSLV/NAME tags + version guards; `NameRecord` + newtypes + smart ctors; `nameRecBytes`/`parseNameRec`/`smpListPUpTo` helpers (no Encoding NameRecord instance); `LookupKey` parser-side cap |
| `Transport.hs` | `namesSMPVersion = 20`; bump current/proxied SMP versions |
| `Server.hs` | Thread `forwarded :: Bool`; `vc SResolver` clauses; whitelist (2132); Resolver branch in `processCommand` (1481); `closeServer` calls `closeNamesEnv`; CSV log (579618); **remove** local `incStat` |
| `Server/Env/STM.hs` | `namesConfig` field; `namesEnv` field; `newEnv` constructs `NamesEnv` with co-location guard |
| `Server/Main.hs` | `[NAMES]` parse: `validateUrl`/`parseEthAddr`/`parseRpcAuth`; `scrubUrl` in logs |
| `Server/Main/Init.hs` | `[NAMES]` block in default INI |
| `Server/Stats.hs` | `incStat` moved here + exported; `NameResolverStats` sub-record + helpers; `rslvStats` field |
| `Server/Prometheus.hs` | `names =` metric block |
| `Server/Names.hs` (new) | Façade re-exports |
| `Server/Names/Resolver.hs` (new) | All resolver types + cache + coalescing + `fetchOnceTimed` + `newNamesEnv[With]` + `closeNamesEnv` |
| `Server/Names/Eth/RPC.hs` (new) | `EthRpcEnv`, `ethCallReal` with bounded body + concurrency semaphore + `Authorization` header |
| `Server/Names/Eth/SNRC.hs` (new) | `EthAddress`, Keccak namehash, bounded ABI (8 invariants), `getRecord` with zero-owner detection |
| `simplexmq.cabal` | Bump `6.6.0.0`; 4 new deps + 4 new modules in `if !flag(client_library)` block |
| `protocol/simplex-messaging.md` | Header version, version-history v20 entry, new "Resolver commands" section |
| `CHANGELOG.md` | v6.6 entry |
## Testing
`tests/SMPNamesTests/` registered in `tests/Test.hs:112151`. Build only when `client_library = False`.
1. **ProtocolEncodingSpec**`nameRecBytes``parseNameRec` round-trip; oversized fields rejected at parse; combined-list cap 8 enforced; negative `nrExpiry` rejected; canonical encoding byte-stable.
2. **MaxSizeSpec** — max `NameRecord` encodes ≤ ~9 KB; `encodeTransmission v ≤ paddedProxiedTLength - 2`; `cbEncrypt` succeeds.
3. **CommandTagSpec**`"RSLV"`/`"NAME"` parse; v < 20 sessions reject `RSLV_` at parameter parser.
4. **ForwardedGateSpec** — direct RSLV → `CMD PROHIBITED`; forwarded RSLV reaches handler.
5. **ForwardedRslvSpec** — RSLV wrapped in PFWD reaches the handler end-to-end. **Test infra cost**: first protocol-level PFWD test; budget for `runProxiedSmpCommand` helper performing `PRXY`/`PKEY`/`PFWD` manually.
6. **CacheSpec** — hit avoids RPC; TTL expiry forces re-fetch; bytes cap evicts before entries cap on large records; concurrent same-key callers issue one RPC; leader exception → all waiters get `Left _`, TMap entry removed; leader async-cancel → cleanup STM still runs.
7. **AbiSpec** — encode/decode against pinned fixtures (`tests/fixtures/snrc/`); QuickCheck fuzz on random buffers ≤ `rpcMaxResponseBytes` must never crash.
8. **NamehashSpec** — Keccak-256 reference vectors; assert Keccak ≠ SHA3-256.
9. **MockRpcSpec** — fake HTTP server; missing → `EthHttpErr`; slow → `TimedOut`; multi-GB body truncated → `EthDecodeErr`. `rpcAuth = AuthBasic` sends correct header.
10. **Uint256OverflowSpec**`expiry > Int64.maxBound``EthDecodeErr`.
11. **ZeroOwnerSpec**`owner = 0x000...000``NotFound`.
12. **StartupGuardSpec**`allowSMPProxy + names.enable` aborts; `allow_dangerous_colocation = on` starts with warning.
13. **UrlValidationSpec** — userinfo/scheme/host/port edge cases; rejects `https://` without `rpc_auth` for non-loopback.
14. **EipChecksumSpec**`parseEthAddr` accepts lower/upper; verifies mixed-case checksum; rejects typos.
15. **AbiBoundsSpec** — each of 8 ABI invariants triggers `EthDecodeErr` without crash/allocation blow-up.
Integration against real Reth+Nimbus mainnet deferred to ops.
## Threat model, scope, coordination
| Actor | Can | Cannot |
|---|---|---|
| Name server | See lookup-key bytes; see query timing; see Eth endpoint URL (operator-self) | See client IP/session; correlate clients across queries |
| Compromised Eth endpoint | Poison this server's cache for one TTL window; see every lookup key the server queries | Bypass two-server agreement (client-side, out of scope) |
| Adversarial client (high-rate unique keys) | Cache-thrash DoS; fill `Manager` connection pool up to `managerConnCount = 8` | Bypass `rpcMaxResponseBytes` or `fetchOnceTimed` |
| Adversarial proxy (slow inner RSLVs) | Block other forwarded commands on that proxy connection up to `rpcTimeoutMs` per miss | Affect other proxy connections |
| Operator with footgun config (https no auth, public Eth RPC) | (rejected at startup, or operator-acknowledged data leak) | — |
Mitigations: caching + coalescing + `rpcTimeoutMs` + `rpcMaxResponseBytes` + `rpcMaxConcurrency`; co-location refused at startup; URL validation; Caddy + auth in front of Reth; Reth's own gas/size caps. Timing side-channels (cache-hit vs miss latency) not mitigated — flagged for post-MVP. State proofs deferred to post-MVP per RFC.
**Cross-repo coordination.** The `simplex-chat` `ep/namespace` branch currently contains only the RFC commit — no agent-side wire-format code yet. This plan's wire format is validated only by simplexmq's own tests until a matching agent PR lands (structurally weak — encoder/decoder bugs are mutually consistent with themselves). Coordinate with the agent-side implementer **before merging** on: exact `NameRecord` field order and types; `LookupKey` namespace-prefix convention; error-code semantics; Part 1 SNRC contract `getRecord` ABI surface.
@@ -1,8 +1,9 @@
# SMP server message storage
# SMP router message storage
## Problem
Currently SMP servers store all queues in server memory. As the traffic grows, so does the number of undelivered messages. What is worse, Haskell is not avoiding heap fragmentation when messages are allocated and then de-allocated - undelivered messages use ByteString and GC cannot move them around, as they use pinned memory.
Currently SMP routers store all queues in router memory. As the traffic grows, so does the number of undelivered messages. What is worse, Haskell is not avoiding heap fragmentation when messages are allocated and then de-allocated - undelivered messages use ByteString and GC cannot move them around, as they use pinned memory.
## Possible solutions
@@ -10,7 +11,7 @@ Currently SMP servers store all queues in server memory. As the traffic grows, s
Move from ByteString to some other primitive to store messages in memory long term, e.g. ShortByteString, or manage allocation/de-allocation of stored messages manually in some other way.
Pros: the simplest solution that avoids substantial re-engineering of the server.
Pros: the simplest solution that avoids substantial re-engineering of the router.
Cons:
- not a long term solution, as memory growth still has limits.
@@ -22,12 +23,12 @@ Use files or RocksDB to store messages.
Pros:
- much lower memory usage.
- no message loss in case of abnormal server termination (important until clients have delivery redundancy).
- no message loss in case of abnormal router termination (important until clients have delivery redundancy).
- this is a long term solution, and at some point it might need to be done anyway.
Cons:
- substantial re-engineering costs and risks.
- metadata privacy. Currently we only save undelivered messages when server is restarted, with this approach all messages will be stored for some time. this argument is limited, as hosting providers of VMs can make memory snapshots too, on the other hand they are harder to analyze than files. On another hand, with this approach messages will be stored for a shorter time.
- metadata privacy. Currently we only save undelivered messages when router is restarted, with this approach all messages will be stored for some time. this argument is limited, as hosting providers of VMs can make memory snapshots too, on the other hand they are harder to analyze than files. On another hand, with this approach messages will be stored for a shorter time.
#### RocksDB and other key-value stores
@@ -67,7 +68,7 @@ queueLogLine =
%s"write_msg=" digits
```
When queue is first requested by the server:
When queue is first requested by the router:
```c
if queue folder exists:
@@ -87,7 +88,7 @@ nextReadMsg = read_msg
open write_file in AppendMode
```
When message is added to the queue (assumes that queue state is loaded to server memory, if not the previous section will be done first):
When message is added to the queue (assumes that queue state is loaded to router memory, if not the previous section will be done first):
```c
if write_msg > max_queue_messages:
@@ -128,7 +129,7 @@ else
nextReadByte = current position in file
```
When message delivery is acknowledged, the read queue needs to be advanced, and possibly switched to read from the current write_queue:
When message delivery is acknowledged, the read queue needs to be advanced, and possibly switched to read from the current write queue:
```c
if nextReadByte == read_byte:
@@ -162,9 +163,9 @@ Most Linux systems use EXT4 filesystem where the file lookup time scales linearl
So storing all queue folders in one folder won't scale.
To solve this problem we could use recipient queue ID in base64url format not as a folder name, but as a folder path, splitting it to path fragments of some length. The number of fragments can be configurable and migration to a different fragment size can be supported as the number of queues on a given server grows.
To solve this problem we could use recipient queue ID in base64url format not as a folder name, but as a folder path, splitting it to path fragments of some length. The number of fragments can be configurable and migration to a different fragment size can be supported as the number of queues on a given router grows.
Currently, queue ID is 24 bytes random number, thus allowing 2^192 possible queue IDs. If we assume that a server must hold 1b queues, it means that we have ~2^162 possible addresses for each existing queue. 24 bytes in base64 is 32 characters that can be split into say 8 fragments with 4 characters each, so that queue folder path for queue with ID `abcdefghijklmnopqrstuvwxyz012345` would be:
Currently, queue ID is 24 bytes random number, thus allowing 2^192 possible queue IDs. If we assume that a router must hold 1b queues, it means that we have ~2^162 possible addresses for each existing queue. 24 bytes in base64 is 32 characters that can be split into say 8 fragments with 4 characters each, so that queue folder path for queue with ID `abcdefghijklmnopqrstuvwxyz012345` would be:
`/var/opt/simplex/messages/abcd/efgh/ijkl/mnop/qrst/uvwx/yz01/2345`
@@ -174,6 +175,6 @@ So we could use an unequal split of path, two letters each and the last being lo
`/var/opt/simplex/messages/ab/cd/ef/ghijklmnopqrstuvwxyz012345`
The first three levels in this case can have 4096 subfolders each, and it gives 68b possible subfolders (64^2^3), so the last level will be sparse in case of 1b queues on the server. So we could make it 4 levels with 2 letters to never think about it, accounting for a large variance of the random numbers distribution:
The first three levels in this case can have 4096 subfolders each, and it gives 68b possible subfolders (64^2^3), so the last level will be sparse in case of 1b queues on the router. So we could make it 4 levels with 2 letters to never think about it, accounting for a large variance of the random numbers distribution:
`/var/opt/simplex/messages/ab/cd/ef/gh/ijklmnopqrstuvwxyz012345`
@@ -1,6 +1,7 @@
# Sharing protocol ports with HTTPS
Some networks block all ports other than web ports, including port 5223 used for SMP protocol by default. Running SMP servers on a common web port 443 would allow them to work on more networks. The servers would need to provide an HTTPS page for browsers (and probes).
Some networks block all ports other than web ports, including port 5223 used for SMP protocol by default. Running SMP routers on a common web port 443 would allow them to work on more networks. The routers would need to provide an HTTPS page for browsers (and probes).
## Problem
@@ -8,7 +9,7 @@ Browsers and tools rely on system CA bundles instead of certificate pinning.
The crypto parameters used by HTTPS are different from what the protocols use.
Public certificate providers like LetsEncrypt can only sign specific types of keys and Ed25519 isn't one of them.
This means a server should distinguish browser and protocol clients and adjust its behavior to match.
This means a router should distinguish browser and protocol clients and adjust its behavior to match.
## Solution
@@ -16,15 +17,15 @@ This means a server should distinguish browser and protocol clients and adjust i
Since LE certificates are only handed out to domain names, TLS client will be sending the SNI.
However client transports are constructed over connected sockets and the SNI wouldn't be present unless explicitly requested.
When a client sends SNI, then it's a browser and a web credentials should be used.
When a client sends SNI, then it's a browser and web credentials should be used.
Otherwise it's a protocol client to be offered the self-signed ca, cert and key.
When a transport colocated with a HTTPS, its ALPN list should be extended with `h2 http/1.1`.
The browsers will send it, and it should be checked before running transport client.
If HTTP ALPN is detected, then the client connection is served with HTTP `Application` instead (the same "server information" page).
If HTTP ALPN is detected, then the client connection is served with HTTP `Application` instead (the same "router information" page).
If some client connects to server IP, doesn't send SNI and doesn't send ALPN, it will look like a pre-handshake client.
In that case a server will send its handshake first.
If some client connects to router IP, doesn't send SNI and doesn't send ALPN, it will look like a pre-handshake client.
In that case a router will send its handshake first.
This can be mitigated by delaying its handshake and letting the probe to issue its HTTP request.
## Implementation plan
@@ -43,7 +44,7 @@ runServer (tcpPort, ATransport t) = do
else runClient serverSignKey t h `runReaderT` env -- performs serverHandshake etc as usual
```
The web app and server live outside, so `runHttp` has to be provided by the `runSMPServer` caller.
The web app and router live outside, so `runHttp` has to be provided by the `runSMPServer` caller.
Additonally, Warp is using its `InternalInfo` object that's scoped to `withII` bracket.
```haskell
@@ -65,11 +66,9 @@ The implementation relies on a few modification to upstream code:
- `warp`: Only the re-export of `serveConnection` is needed.
Unfortunately the most recent `warp` version can't be used right away due to dependency cascade around `http-5` and `auto-update-2`.
So a fork containing the backported re-export has to be used until the dependencies are refreshed.
### TLS.ServerParams
When a server has port sharing enabled, a new set of TLS params is loaded and combined with transport params:
When a router has port sharing enabled, a new set of TLS params is loaded and combined with transport params:
```haskell
newEnv config = do
@@ -129,7 +128,7 @@ key: /etc/opt/simplex/web.key
# key: /etc/letsencrypt/live/smp.hostname.tld/privkey.pem
```
When `TRANSPORT.port` matches `WEB.https` the transport server becomes shared.
When `TRANSPORT.port` matches `WEB.https` the transport router becomes shared.
Perhaps a more desirable option would be explicit configuration resulting in additional transported to run:
@@ -148,16 +147,16 @@ key: /etc/opt/simplex/web.key
## Caveats
Serving static files and the protocols togother may pose a problem for those who currently use dedicated web servers as they should switch to embedded http handlers.
Serving static files and the protocols together may pose a problem for those who currently use dedicated web servers as they should switch to embedded http handlers.
As before, using embedded HTTP server is increasing attack surface.
Users who want to run everything on a single host will have to add and extra IP address and bind servers to specific IPs instead of 0.0.0.0.
An amalgamated server binary can be provided that would contain both SMP and XFTP servers, where transport will dispatch connections by handshake ALPN.
Users who want to run everything on a single host will have to add an extra IP address and bind routers to specific IPs instead of 0.0.0.0.
An amalgamated router binary can be provided that would contain both SMP and XFTP routers, where transport will dispatch connections by handshake ALPN.
## Alternative: Use transports routable with reverse-proxies
An "industrial" reverse proxy may do the ALPN routing, serving HTTP by itself and delegating `smp` and `xftp` to protocol servers.
Same with the `websockets`.
Since this in effect does TLS termination, the protocol servers will have to rely on credentials from protocol handshakes.
Since this in effect does TLS termination, the protocol routers will have to rely on credentials from protocol handshakes.
@@ -1,8 +1,9 @@
# Expiring messages in journal storage
## Problem
The journal storage servers recently migrated to do not delete delivered or expired messages, they only update pointers to journal file lines. The messages are actually deleted when the whole journal file is deleted (when fully deleted or fully expired).
The journal storage routers recently migrated to do not delete delivered or expired messages, they only update pointers to journal file lines. The messages are actually deleted when the whole journal file is deleted (when fully deleted or fully expired).
The problem is that in case the queue stops receiving the new messages then writing of messages won't switch to the new journal file, and the current journal file containing delivered or expired messages would never be deleted.
@@ -1,3 +1,4 @@
# Fix subQ deadlock: blocking writeTBQueue inside connLock
## Problem
+331 -73
View File
@@ -1,4 +1,4 @@
Version 5, 2024-06-22
Version 7, 2025-01-24
# SMP agent protocol - duplex communication over SMP protocol
@@ -6,9 +6,10 @@ Version 5, 2024-06-22
- [Abstract](#abstract)
- [SMP agent](#smp-agent)
- [SMP servers management](#smp-servers-management)
- [SMP routers management](#smp-routers-management)
- [SMP agent protocol scope](#smp-agent-protocol-scope)
- [Duplex connection procedure](#duplex-connection-procedure)
- [Fast duplex connection procedure](#fast-duplex-connection-procedure)
- [Contact addresses](#contact-addresses)
- [Communication between SMP agents](#communication-between-smp-agents)
- [Message syntax](#messages-between-smp-agents)
@@ -20,41 +21,58 @@ Version 5, 2024-06-22
- [Rotating messaging queue](#rotating-messaging-queue)
- [End-to-end encryption](#end-to-end-encryption)
- [Connection link: 1-time invitation and contact address](#connection-link-1-time-invitation-and-contact-address)
- [Appendix A: SMP agent API](#smp-agent-api)
- [Full connection link syntax](#full-connection-link-syntax)
- [Short connection link syntax](#short-connection-link-syntax)
- [Short links](#short-links)
- [Link key derivation](#link-key-derivation)
- [Link data encryption](#link-data-encryption)
- [Short link resolution](#short-link-resolution)
- [Link data management](#link-data-management)
- [Appendix A: SMP agent API](#appendix-a-smp-agent-api)
- [API functions](#api-functions)
- [API events](#api-events)
## Abstract
The purpose of SMP agent protocol is to define the syntax and the semantics of communications between the client and the agent that connects to [SMP](./simplex-messaging.md) servers.
The purpose of SMP agent protocol is to define the syntax and the semantics of communications between the client and the agent that connects to [SMP](./simplex-messaging.md) routers.
It provides:
- API to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the servers location from the users of the agent protocol.
- API to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the routers location from the users of the agent protocol.
- management of E2E encryption between SMP agents, generating ephemeral asymmetric keys for each connection.
- SMP command authentication on SMP servers, generating ephemeral keys for each SMP queue.
- TCP/TLS transport handshake with SMP servers.
- SMP command authentication on SMP routers, generating ephemeral keys for each SMP queue.
- TCP/TLS transport handshake with SMP routers.
- validation of message integrity.
SMP agent API provides no security between the agent and the client - it is assumed that the agent is executed in the trusted and secure environment, via the agent library, when the agent logic is included directly into the client application - [SimpleX Chat for terminal](https://github.com/simplex-chat/simplex-chat) uses this approach.
This document describes SMP agent protocol version 7. The version history:
- v1: initial version
- v2: duplex handshake - allows including reply queue(s) in the initial confirmation
- v3: ratchet sync - supports re-negotiating double ratchet encryption
- v4: delivery receipts - supports acknowledging message delivery to the sender
- v5: post-quantum - supports post-quantum key exchange in double ratchet (PQDR)
- v6: sender auth key - supports sender authentication key in confirmations
- v7: ratchet on confirmation - initializes double ratchet during confirmation
## SMP agent
SMP agents communicate with each other via SMP servers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the API calls used by the client applications. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections").
SMP agents communicate with each other via SMP routers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the API calls used by the client applications. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections").
The agent must have a persistent storage to manage the states of known connections and of the client-side information of SMP queues that each connection consists of, and also the buffer of the most recent sent and received messages. The number of the messages that should be stored is implementation specific, depending on the error management approach that the agent implements; at the very least the agent must store the hashes and IDs of the last received and sent messages.
## SMP servers management
## SMP routers management
SMP agent API does not use the addresses of the SMP servers that the agent will use to create and use the connections (excluding the server address in queue URIs used in JOIN command). The list of the servers is a part of the agent configuration and can be dynamically changed by the agent implementation:
SMP agent API does not use the addresses of the SMP routers that the agent will use to create and use the connections (excluding the router address in queue URIs used in JOIN command). The list of the routers is a part of the agent configuration and can be dynamically changed by the agent implementation:
- by the client applications via any API that is outside of scope of this protocol.
- by the agents themselves based on availability and latency of the configured servers.
- by the agents themselves based on availability and latency of the configured routers.
## SMP agent protocol scope
SMP agent protocol has 2 main parts:
- the messages that SMP agents exchange with each other in order to:
- negotiate establishing unidirectional (simplex) encrypted queues on SMP servers.
- negotiate establishing unidirectional (simplex) encrypted queues on SMP routers.
- exchange client messages and delivery notifications, providing sequential message IDs and message integrity (by including the hash of the previous message).
- re-negotiate messaging queues to use and connection e2e encryption.
- the messages that the clients of SMP agents should send out-of-band (as pre-shared "invitation" including queue URIs) to protect [E2E encryption][1] from active attacks ([MITM attacks][2]).
@@ -67,40 +85,40 @@ SMP agent protocol has 2 main parts:
![Duplex connection procedure](./diagrams/duplex-messaging/duplex-creating.svg)
The procedure of establishing a duplex connection is explained on the example of Alice and Bob creating a bi-directional connection consisting of two unidirectional (simplex) queues, using SMP agents (A and B) to facilitate it, and two different SMP servers (which could be the same server). It is shown on the diagram above and has these steps:
The procedure of establishing a duplex connection is explained on the example of Alice and Bob creating a bi-directional connection consisting of two unidirectional (simplex) queues, using SMP agents (A and B) to facilitate it, and two different SMP routers (which could be the same router). It is shown on the diagram above and has these steps:
1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function.
2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md) `NEW` command) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
2. Agent A creates an SMP queue on the router (using [SMP protocol](./simplex-messaging.md) `NEW` command) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
3. Alice sends the [connection link](#connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code.
4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection.
5. Agent B creates Bob's SMP reply queue with SMP server `NEW` command.
6. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
6. Alice confirms and continues the connection:
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`.
5. Agent B creates Bob's SMP reply queue with SMP router `NEW` command.
6. Agent B confirms the connection: sends an "SMP confirmation" with SMP router `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
7. Alice confirms and continues the connection:
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP router `MSG`.
- Agent A notifies Alice sending `CONF` notification with Bob's info.
- Alice allows connection to continue with agent `allowConnection` api function.
- Agent A secures the queue with SMP server `KEY` command.
- Agent A secures the queue with SMP router `KEY` command.
- Agent A sends SMP confirmation with ephemeral sender key, ephemeral public encryption key and profile (but without reply queue).
7. Agent B confirms the connection:
8. Agent B confirms the connection:
- receives the confirmation.
- sends the notification `INFO` with Alice's information to Bob.
- secures SMP queue that it sent to Alice in the first confirmation with SMP `KEY` command .
- sends `HELLO` message via SMP `SEND` command. This confirms that the reply queue is secured and also validates that Agent A secured the first SMP queue
8. Agent A notifies Alice.
9. Agent A notifies Alice.
- receives `HELLO` message from Agent B.
- sends `HELLO` message to Agent B via SMP `SEND` command.
- sends `CON` notification to Alice, confirming that the connection is established.
9. Agent B notifies Bob.
10. Agent B notifies Bob.
- Once Agent B receives `HELLO` from Agent A, it sends to Bob `CON` notification as well.
At this point the duplex connection between Alice and Bob is established, they can use `SEND` command to send messages. The diagram also shows how the connection status changes for both parties, where the first part is the status of the SMP queue to receive messages, and the second part - the status of the queue to send messages.
The most communication happens between the agents and servers, from the point of view of Alice and Bob there are 4 steps (not including notifications):
The most communication happens between the agents and routers, from the point of view of Alice and Bob there are 4 steps (not including notifications):
1. Alice requests a new connection with `createConnection` agent API function and receives the connection link.
2. Alice passes connection link out-of-band to Bob.
3. Bob accepts the connection with `joinConnection` agent API function with the connection link to his agent.
4. Alice accepts the connection with `ACPT` agent API function.
4. Alice accepts the connection with `allowConnection` agent API function.
5. Both parties receive `CON` notification once duplex connection is established.
Clients SHOULD support establishing duplex connection asynchronously (when parties are intermittently offline) by persisting intermediate states and resuming SMP queue subscriptions.
@@ -118,14 +136,14 @@ Faster duplex connection process is possible with the `SKEY` command added in v9
![Fast duplex connection procedure](./diagrams/duplex-messaging/duplex-creating-fast.svg)
1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function
2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md) `NEW` command with the flag allowing the sender to secure the queue) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
2. Agent A creates an SMP queue on the router (using [SMP protocol](./simplex-messaging.md) `NEW` command with the flag allowing the sender to secure the queue) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
3. Alice sends the [connection link](connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code. This link contains the flag that the queue can be secured by the sender.
4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection.
5. Agent B secures Alice's queue with SMP command `SKEY` - this command can be proxied.
6. Agent B creates Bob's SMP reply queue with SMP server `NEW` command (with the flag allowing the sender to secure the queue).
7. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
6. Agent B creates Bob's SMP reply queue with SMP router `NEW` command (with the flag allowing the sender to secure the queue).
7. Agent B confirms the connection: sends an "SMP confirmation" with SMP router `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
8. Alice confirms the connection:
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`.
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP router `MSG`.
- Agent A notifies Alice sending `CONF` notification with Bob's info (that indicates that Agent B already secured the queue).
- Alice allows connection to continue with agent `allowConnection` api function.
- Agent A secures Bob's queue with SMP command `SKEY`.
@@ -140,11 +158,11 @@ Faster duplex connection process is possible with the `SKEY` command added in v9
SMP agents support creating a special type of connection - a contact address - that allows to connect to multiple network users who can send connection requests by sending 1-time connection links to the message queue.
This connection address uses a messaging queue on SMP server to receive invitations to connect - see `agentInvitation` message below. Once connection request is accepted, a new connection is created and the address itself is no longer used to send the messages - deleting this address does not disrupt the connections that were created via it.
This connection address uses a messaging queue on SMP router to receive invitations to connect - see `agentInvitation` message below. Once connection request is accepted, a new connection is created and the address itself is no longer used to send the messages - deleting this address does not disrupt the connections that were created via it.
## Communication between SMP agents
To establish duplex connections and to send messages on behalf of their clients, SMP agents communicate via SMP servers.
To establish duplex connections and to send messages on behalf of their clients, SMP agents communicate via SMP routers.
Agents use SMP message client body (the part of the SMP message after header - see [SMP protocol](./simplex-messaging.md)) to transmit agent client messages and exchange messages between each other.
@@ -152,13 +170,13 @@ These messages are encrypted with per-queue shared secret using NaCL crypto_box
- `agentConfirmation` - used when confirming SMP queues, contains connection information encrypted with double ratchet. This envelope can only contain `agentConnInfo` or `agentConnInfoReply` encrypted with double ratchet.
- `agentMsgEnvelope` - contains different agent messages encrypted with double ratchet, as defined in `agentMessage`.
- `agentInvitation` - sent to SMP queue that is used as contact address, does not use double ratchet.
- `agentRatchetKey` - used to re-negotiate double ratchet encryption - can contain additional information in `agentRatchetKey`.
- `agentRatchetKey` - used to re-negotiate double ratchet encryption - can contain additional information in `agentRatchetInfo`.
```abnf
decryptedSMPClientMessage = agentConfirmation / agentMsgEnvelope / agentInvitation / agentRatchetKey
agentConfirmation = agentVersion %s"C" ("0" / "1" sndE2EEncryptionParams) encConnInfo
agentVersion = 2*2 OCTET
sndE2EEncryptionParams = TODO
sndE2EEncryptionParams = <sender E2E ratchet parameters, see pqdr.md>
encConnInfo = doubleRatchetEncryptedMessage
agentMsgEnvelope = agentVersion %s"M" encAgentMessage
@@ -166,13 +184,25 @@ encAgentMessage = doubleRatchetEncryptedMessage
agentInvitation = agentVersion %s"I" connReqLength connReq connInfo
connReqLength = 2*2 OCTET ; Word16
connReq = *OCTET ; URI text encoding of connection link, length given by connReqLength
connInfo = *OCTET ; opaque connection information (remaining bytes)
agentRatchetKey = agentVersion %s"R" rcvE2EEncryptionParams agentRatchetInfo
rcvE2EEncryptionParams = TODO
agentRatchetKey = agentVersion %s"R" rcvE2EEncryptionParams ratchetKeyInfo
rcvE2EEncryptionParams = <receiver E2E ratchet parameters, see pqdr.md>
ratchetKeyInfo = *OCTET ; additional ratchet renegotiation info (remaining bytes)
doubleRatchetEncryptedMessage = TODO
doubleRatchetEncryptedMessage = <double ratchet encrypted message, see pqdr.md>
```
The maximum size of the encrypted connection info and agent message depend on whether post-quantum key exchange is used:
| Constant | PQ on | PQ off |
|----------|-------|--------|
| `e2eEncConnInfoLength` | 11106 | 14832 |
| `e2eEncAgentMsgLength` | 13618 | 15840 |
The PQ-on sizes are smaller because the ratchet header and reply link include larger PQ keys (SNTRUP761).
This syntax of decrypted SMP client message body is defined by `decryptedAgentMessage` below.
Decrypted SMP message client body can be one of 4 types:
@@ -182,14 +212,15 @@ Decrypted SMP message client body can be one of 4 types:
- `agentMessage` - all other agent messages.
`agentMessage` contains these parts:
- `agentMsgHeader` - agent message header that contains sequential agent message ID for a particular SMP queue, agent timestamp (ISO8601) and the hash of the previous message.
- `agentMsgHeader` - agent message header that contains sequential agent message ID for a particular SMP queue and the hash of the previous message.
- `aMessage` - a command/message to the other SMP agent:
- to confirm the connection (`HELLO`).
- to send and to confirm reception of user messages (`A_MSG`, `A_RCVD`).
- to confirm that the new double ratchet encryption is agreed (`EREADY`).
- to notify another party that it can continue sending messages after queue capacity was exceeded (`A_QCONT`).
- to manage SMP queue rotation (`QADD`, `QKEY`, `QUSE`, `QTEST`).
- `msgPadding` - an optional message padding to make all SMP messages have constant size, to prevent servers from observing the actual message size. The only case the message padding can be absent is when the message has exactly the maximum size, in all other cases the message MUST be padded to a fixed size.
The encoded `agentMessage` is padded to a fixed size by the double ratchet encryption layer (see [ratchet message wire format](./pqdr.md#ratchet-message-wire-format)) to make all SMP messages have constant size, preventing routers from observing the actual message size.
### Messages between SMP agents
@@ -200,9 +231,11 @@ decryptedAgentMessage = agentConnInfo / agentConnInfoReply / agentRatchetInfo /
agentConnInfo = %s"I" connInfo
connInfo = *OCTET
agentConnInfoReply = %s"D" smpQueues connInfo
smpQueues = length 1*newQueueInfo ; NonEmpty list of reply queues
agentRatchetInfo = %s"R" ratchetInfo
ratchetInfo = *OCTET
agentMessage = %s"M" agentMsgHeader aMessage msgPadding
agentMessage = %s"M" agentMsgHeader aMessage
agentMsgHeader = agentMsgId prevMsgHash
agentMsgId = 8*8 OCTET ; Int64
prevMsgHash = shortString
@@ -213,10 +246,13 @@ aMessage = HELLO / A_MSG / A_RCVD / EREADY / A_QCONT /
HELLO = %s"H"
A_MSG = %s"M" userMsgBody
userMsgBody = *OCTET
userMsgBody = *OCTET ; remaining bytes
A_RCVD = %s"V" msgReceipt
A_RCVD = %s"V" msgReceipts
msgReceipts = length 1*msgReceipt ; NonEmpty list
msgReceipt = agentMsgId msgHash rcptLength rcptInfo
msgHash = shortString
rcptInfo = *OCTET ; opaque receipt info, length given by rcptLength (Word16)
EREADY = %s"E" agentMsgId
@@ -224,14 +260,14 @@ A_QCONT = %s"QC" sndQueueAddr
QADD = %s"QA" sndQueues
sndQueues = length 1*(newQueueUri replacedSndQueue)
newQueueUri = clientVRange smpServer senderId dhPublicKey [sndSecure]
newQueueUri = clientVRange smpRouter senderId dhPublicKey [queueMode]
dhPublicKey = length x509encoded
sndSecure = "T"
queueMode = %s"M" / %s"C" ; M - messaging (sender can secure), C - contact
replacedSndQueue = "0" / "1" sndQueueAddr
QKEY = %s"QK" sndQueueKeys
sndQueueKeys = length 1*(newQueueInfo senderKey)
newQueueInfo = version smpServer senderId dhPublicKey [sndSecure]
newQueueInfo = version smpRouter senderId dhPublicKey [queueMode]
senderKey = length x509encoded
QUSE = %s"QU" sndQueuesReady
@@ -241,8 +277,8 @@ primary = %s"T" / %s"F"
QTEST = %s"QT" sndQueueAddrs
sndQueueAddrs = length 1*sndQueueAddr
sndQueueAddr = smpServer senderId
smpServer = hosts port keyHash
sndQueueAddr = smpRouter senderId
smpRouter = hosts port keyHash
hosts = length 1*host
host = shortString
port = shortString
@@ -252,7 +288,6 @@ senderId = shortString
clientVRange = version version
version = 2*2 OCTET
msgPadding = *OCTET
rcptLength = 2*2 OCTET
shortString = length *OCTET
length = 1*1 OCTET
@@ -266,11 +301,11 @@ This message is not used with [fast duplex connection](#fast-duplex-connection-p
#### A_MSG message
This is the agent envelope used to send client messages once the connection is established. This is different from the MSG sent by SMP server to the agent and MSG event from SMP agent to the client that are sent in different contexts.
This is the agent envelope used to send client messages once the connection is established. This is different from the MSG sent by SMP router to the agent and MSG event from SMP agent to the client that are sent in different contexts.
#### A_RCVD message
This message is sent to confirm the client message reception. It includes received message number and message hash.
This message is sent to confirm the client message reception. It includes a list of message receipts, each containing the received message number, message hash and receipt info.
#### EREADY message
@@ -282,7 +317,7 @@ This message is sent to notify the sender client that it can continue sending th
### Rotating messaging queue
SMP agents SHOULD support 4 messages to rotate message reception to another messaging server:
SMP agents SHOULD support 4 messages to rotate message reception to another messaging router:
`QADD`: add the new queue address(es) to the connection - sent by the client that initiates rotation.
`QKEY`: pass sender's key via existing connection (SMP confirmation message will not be used, to avoid the same "race" of the initial key exchange that would create the risk of intercepting the queue for the attacker) - sent by the client accepting the rotation
`QUSE`: instruct the sender to use the new queue with sender's queue ID as parameter. From this point some messages can be sent to both the new queue and the old queue.
@@ -345,31 +380,191 @@ To summarize, the upgrade to DH+KEM secret happens in a sent message that has PQ
Connection links are generated by SMP agent in response to `createConnection` api call, used by another party user with `joinConnection` api, and then another connection link is sent by the agent in `agentConnInfoReply` and used by the first party agent to connect to the reply queue (the second part of the process is invisible to the users).
Connection link syntax:
### Full connection link syntax
```
connectionLink = connectionScheme "/" connLinkType "#/?smp=" smpQueues "&e2e=" e2eEncryption
connectionLink = connectionScheme "/" connLinkType "#/?v=" versionRange "&smp=" smpQueues ["&e2e=" e2eEncryption] ["&data=" clientData]
connLinkType = %s"invitation" / %s"contact"
connectionScheme = (%s"https://" clientAppServer) | %s"simplex:"
connectionScheme = (%s"https://" clientAppServer) / %s"simplex:"
clientAppServer = hostname [ ":" port ]
; client app server, e.g. simplex.chat
e2eEncryption = encryptionScheme ":" publicKey
encryptionScheme = %s"rsa" ; end-to-end encryption and key exchange protocols,
; the current hybrid encryption scheme (RSA-OAEP/AES-256-GCM-SHA256)
; will be replaced with double ratchet protocol and DH key exchange.
publicKey = <base64url X509 SPKI key encoding>
smpQueues = smpQueue [ "," 1*smpQueue ] ; SMP queues for the connection
versionRange = 1*DIGIT / 1*DIGIT "-" 1*DIGIT ; agent version range
e2eEncryption = <e2e encryption parameters for double ratchet>
smpQueues = smpQueue *(";" smpQueue) ; SMP queues for the connection (semicolon-separated)
smpQueue = <URL-encoded queueURI defined in SMP protocol>
clientData = <URL-encoded application-specific data>
```
All parameters are passed via URI hash to avoid sending them to the server (in case "https" scheme is used) - they can be used by the client-side code and processed by the client application. Parameters `smp` and `e2e` can be present in any order, any unknown additional parameters SHOULD be ignored.
All parameters are passed via URI hash to avoid sending them to the router (in case "https" scheme is used) - they can be used by the client-side code and processed by the client application. Parameters can be present in any order, any unknown additional parameters SHOULD be ignored.
`clientAppServer` is not an SMP server - it is a server that shows the instruction on how to download the client app that will connect using this connection link. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device.
`clientAppServer` is not an SMP router - it is a server that shows the instruction on how to download the client app that will connect using this connection link. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device.
"simplex" URI scheme in `connectionProtocol` can be used instead of client app server, to connect without creating any web traffic. Client apps MUST support this URI scheme.
"simplex" URI scheme in `connectionProtocol` can be used instead of client app router, to connect without creating any web traffic. Client apps MUST support this URI scheme.
See SMP protocol [out-of-band messages](./simplex-messaging.md#out-of-band-messages) for syntax of `queueURI`.
### Short connection link syntax
Short links provide a more compact representation by storing connection data on the router:
```
shortLink = shortLinkScheme "/" linkType "#" [linkId "/"] linkKey ["?" shortLinkParams]
shortLinkScheme = %s"simplex:" / (%s"https://" serverHost)
linkType = %s"i" / contactType ; i - invitation, or contact type
contactType = %s"a" / %s"c" / %s"g" / %s"r" ; a - contact, c - channel, g - group, r - relay
linkId = base64url ; only for invitation links
linkKey = base64url ; SHA3-256 hash of fixed data, used to decrypt link data
shortLinkParams = hostParam ["&" portParam] ["&" keyHashParam]
hostParam = %s"h=" hostList
hostList = host *("," host)
portParam = %s"p=" port
keyHashParam = %s"c=" base64url ; router certificate fingerprint
```
Contact types:
- `a` (CCTContact) - direct contact connection
- `c` (CCTChannel) - channel connection
- `g` (CCTGroup) - group connection
- `r` (CCTRelay) - relay connection
Short links can use either the `simplex:` scheme or `https://` with a router hostname. When using the simplex scheme, router information is included in query parameters.
## Short links
Short links provide a compact representation of connection links by storing encrypted connection data on the SMP router. The link key in the URI fragment (after `#`) is never sent to the router, ensuring the router cannot decrypt the stored connection data.
### Link key derivation
The link key is derived from the fixed link data using SHA3-256 hash function:
```
linkKey = SHA3-256(fixedLinkData)
```
The fixed link data includes:
- Agent version range
- Root public key (Ed25519) for signing
- SMP queue connection request (router, queue IDs, encryption keys)
- Optional link entity ID
For contact links, the link ID and encryption key are derived from the link key using HKDF:
```
(linkId, encryptionKey) = HKDF(info="SimpleXContactLink", key=linkKey, outputLen=56)
; linkId = first 24 bytes, encryptionKey = remaining 32 bytes
```
For invitation links, the link ID is stored separately (usually included in the URI), and only the encryption key is derived:
```
encryptionKey = HKDF(info="SimpleXInvLink", key=linkKey, outputLen=32)
```
### Link data encryption
Link data stored on the router consists of two encrypted parts: fixed data and user data. Both are encrypted using NaCl secret_box (XSalsa20-Poly1305) with the derived encryption key:
```abnf
queueLinkData = encFixedData encUserData
encFixedData = largeString ; encrypted padded(signedFixedData, 2008)
encUserData = largeString ; encrypted padded(signedUserData, 13784)
signedFixedData = signature fixedData
signedUserData = signature userData
signature = length 64*64 OCTET ; Ed25519 signature
fixedData = agentVersionRange rootKey linkConnReq [linkEntityId]
agentVersionRange = version version ; min and max agent protocol version
version = 2*2 OCTET
rootKey = length x509encoded ; Ed25519 public key
linkConnReq = invitationConnReq / contactConnReq ; binary encoding of connection request
invitationConnReq = %s"I" connReqData e2eRatchetParams
contactConnReq = %s"C" connReqData
linkEntityId = shortString
userData = invitationLinkData / contactLinkData
invitationLinkData = %s"I" agentVersionRange userLinkData
contactLinkData = %s"C" agentVersionRange userContactData
userLinkData = shortString / (%xFF largeString) ; opaque application data (e.g., user profile)
; shortString length byte 0x00-0xFE (max 254 bytes); 0xFF is reserved as largeString sentinel
userContactData = direct ownersList relaysList userLinkData
direct = %s"T" / %s"F" ; whether direct connection via connReq is allowed
ownersList = length *ownerAuth
ownerAuth = shortString ; length-prefixed encoding of (ownerId ownerKey authOwnerSig)
ownerId = shortString ; application-specific owner ID (e.g., MemberId)
ownerKey = length x509encoded ; Ed25519 public key
authOwnerSig = length 64*64 OCTET ; Ed25519 signature of (ownerId || ownerKey) by previous owner
relaysList = length *connShortLink ; alternative relay short links
; Binary encoding of connection request (used in linkConnReq)
connReqData = agentVersionRange smpQueueUris clientData
smpQueueUris = length 1*smpQueueUri
clientData = %s"0" / (%s"1" largeString) ; Maybe (Large ByteString)
smpQueueUri = smpClientVersionRange smpServer senderId smpDhPublicKey [queueMode]
smpClientVersionRange = version version ; min and max SMP client versions
smpServer = hosts port serverKeyHash
hosts = length 1*host
host = shortString ; text-encoded hostname or IP address
port = shortString ; text-encoded port number
serverKeyHash = shortString ; CA certificate fingerprint
senderId = shortString ; queue sender ID
smpDhPublicKey = length x509encoded ; X25519 DH public key
queueMode = %s"M" / %s"C" ; messaging or contact (version-dependent trailing field)
e2eRatchetParams = e2eVersionRange e2eDhKey e2eDhKey kemParams
e2eVersionRange = version version ; min and max e2e encryption versions
e2eDhKey = length x509encoded ; X448 DH public key
kemParams = %s"0" / (%s"1" ratchetKEMParams)
ratchetKEMParams = %s"P" kemPublicKey / %s"A" kemCiphertext kemPublicKey
kemPublicKey = largeString ; sntrup761 public key
kemCiphertext = largeString ; sntrup761 ciphertext
; Binary encoding of short link (used in relaysList)
connShortLink = invShortLink / contactShortLink
invShortLink = %s"I" smpServer linkId linkKey
contactShortLink = %s"C" contactConnType smpServer linkKey
contactConnType = %s"A" / %s"C" / %s"G" / %s"R" ; contact / channel / group / relay
linkId = shortString
linkKey = shortString
x509encoded = *OCTET ; DER-encoded X.509 SubjectPublicKeyInfo
largeString = 2*2 OCTET *OCTET ; Word16 length prefix
length = 1*1 OCTET
shortString = length *OCTET
```
The fixed data is signed with the root key and its hash becomes the link key. The user data is signed either with the root key (for invitations) or with an owner key (for contact addresses).
### Short link resolution
When a user receives a short link, the agent resolves it as follows:
1. Extract the link key from the URI fragment
2. Send `LGET` command to the SMP router with the link ID
3. Receive encrypted link data from the router
4. Decrypt the link data using the link key
5. Extract the full connection information (SMP queue URI, encryption keys, profile)
6. Proceed with the standard connection procedure using `joinConnection`
For invitation links, the `LKEY` command is used to set the sender key when getting link data. Repeated `LKEY` would require using the same key.
### Link data management
The recipient who created the queue can manage the short link data:
- **LSET** - Set or update the link data associated with a queue. This is used when creating a short link or updating the user data (e.g., profile changes).
- **LDEL** - Delete the link data from the router. This effectively invalidates the short link.
Short links support different connection modes:
- **invitation** - One-time invitation links that can only be used once
- **contact** - Reusable contact address links that can be used multiple times
For contact addresses, the link data includes additional information about the contact type:
- **contact** - Direct contact connection
- **channel** - Channel connection
- **group** - Group connection
- **relay** - Relay connection
The agent maintains the link data and updates it when connection parameters change, ensuring short links remain valid and reflect current connection information.
## Appendix A: SMP agent API
The exact specification of agent library API and of the events that the agent sends to the client application is out of scope of the protocol specification.
@@ -380,7 +575,7 @@ The list of some of the API functions and events below is supported by the refer
The list of APIs below is not exhaustive and provided for information only. Please consult the source code for more information.
#### Create conection
#### Create connection
`createConnection` api is used to create a connection - it returns the connection link that should be sent out-of-band to another protocol user (the joining party). It should be used by the client of the agent that initiates creating a duplex connection (the initiating party).
@@ -408,13 +603,13 @@ Client can `acceptContact` and `rejectContact`, with `OK` and `ERR` events in ca
#### Send message
`sendMessage` api is always asynchronous. The api call returns message ID, `SENT` event once the message is sent to the server, `MWARN` event in case of temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client) and `MERR` in case of permanent delivery failure.
`sendMessage` api is always asynchronous. The api call returns message ID, `SENT` event once the message is sent to the router, `MWARN` event in case of temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client) and `MERR` in case of permanent delivery failure.
#### Acknowledge received message
Messages are delivered to the client application via `MSG` event.
Client application must always `ackMessage` to receive the next one - failure to call it in reference implementation will prevent the delivery of subsequent messages until the client reconnects to the server.
Client application must always `ackMessage` to receive the next one - failure to call it in reference implementation will prevent the delivery of subsequent messages until the client reconnects to the router.
This api is also used to acknowledge message delivery to the sending party - that party client application will receive `RCVD` event.
@@ -426,9 +621,17 @@ This api is also used to acknowledge message delivery to the sending party - tha
`getNotificationMessage` is used by push notification subsystem of the client application to receive the message from a specific messaging queue mentioned in the notification. The client application would receive `MSG` and any other events from the agent, and then `MSGNTF` event once the message related to this notification is received.
#### Rotate message queue to another server
#### Set short link data
`switchConnection` api is used to rotate connection queues to another messaging server.
`setConnectionLink` api (`LSET` command) is used to set or update short link data associated with a contact address queue. Returns `LINK` event with the short link URI.
#### Get short link data
`getConnectionLink` api (`LGET` command) is used to retrieve and decrypt the short link data from the router. Returns `LDATA` event with the decrypted link data.
#### Rotate message queue to another router
`switchConnection` api is used to rotate connection queues to another messaging router.
#### Renegotiate e2e encryption
@@ -436,7 +639,7 @@ This api is also used to acknowledge message delivery to the sending party - tha
#### Delete connection
`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQ` and `DEL_CONN` events.
`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQS` and `DEL_CONNS` events.
#### Suspend connection
@@ -451,25 +654,80 @@ Agent API uses these events dispatch to notify client application about events r
- `INFO` - information from the party that initiated the connection with `createConnection` sent to the party accepting the connection with `joinConnection`.
- `CON` - notification that connection is established sent to both parties of the connection.
- `END` - notification that connection subscription is terminated when another client subscribed to the same messaging queue.
- `DOWN` - notification that connection server is temporarily unavailable.
- `UP` - notification that the subscriptions made in the current client session are resumed after the server became available.
- `DOWN` - notification that connection router is temporarily unavailable.
- `UP` - notification that the subscriptions made in the current client session are resumed after the router became available.
- `SWITCH` - notification about queue rotation process.
- `RSYNC` - notification about e2e encryption re-negotiation process.
- `SENT` - notification to confirm that the message was delivered to at least one of SMP servers. This notification contains the same message ID as returned to `sendMessage` api. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session.
- `SENT` - notification to confirm that the message was delivered to at least one of SMP routers. This notification contains the same message ID as returned to `sendMessage` api. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session.
- `MWARN` - temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client).
- `MERR` - notification about permanent message delivery failure.
- `MERRS` - notification about permanent message delivery failure for multiple messages (e.g., when multiple messages expire).
- `MSG` - sent when agent receives the message from the SMP server.
- `MSG` - sent when agent receives the message from the SMP router.
- `MSGNTF` - sent after agent received and processed the message referenced in the push notification.
- `RCVD` - notification confirming message receipt by another party.
- `QCONT` - notification that the agent continued sending messages after queue capacity was exceeded and recipient received all messages.
- `DEL_RCVQ` - confirmation that message queue was deleted.
- `DEL_CONN` - confirmation that connection was deleted.
- `LINK` - short link URI created or updated for a contact address.
- `LDATA` - decrypted short link data received from the router.
- `DELD` - notification that the connection was deleted.
- `JOINED` - notification that a member joined via a contact address.
- `STAT` - connection statistics event.
- `DEL_RCVQS` - confirmation that receiver message queues were deleted.
- `DEL_CONNS` - confirmation that connections were deleted.
- `OK` - confirmation that asynchronous api call was successful.
- `ERR` - error of asynchronous api call or some other error event.
This list of events is not exhaustive and provided for information only. Please consult the source code for more information.
## Threat model
This threat model complements SimpleX Messaging Protocol [threat model](./security.md#threat-model) with agent-level concerns: duplex connections, end-to-end encryption with [post-quantum double ratchet](./pqdr.md), message integrity, connection establishment and queue rotation. Only additional properties not covered in the SMP threat model are listed below.
#### Additional global assumptions
- The connection link is shared via a trusted out-of-band channel.
- Both agents support post-quantum double ratchet (PQDR).
#### A passive adversary
*cannot:*
- learn the contents of packets, which are additionally encrypted with the double ratchet independently from per-queue encryption.
#### Destination router (chosen by the receiving client application)
*can:*
- correlate queues belonging to the same duplex connection when queue rotation creates a new queue on the same router.
- when both peers of a connection chose the same router, correlate the two directions of the duplex connection.
*cannot:*
- compromise end-to-end encryption even with full access to the per-queue NaCl DH secret.
- correlate queues belonging to the same connection after queue rotation to a different router.
#### An attacker who obtained a client application's (decrypted) database
*can:*
- learn the full communication graph: all communication peers, associated router addresses, and queue identifiers.
*cannot:*
- decrypt future messages once the client application resumes communication and the double ratchet completes a new ratchet step, provided PQDR is active.
#### A communication peer
*can:*
- send malformed agent messages that may affect the client application processing them.
- skip message IDs, causing the recipient to generate and store excessive intermediate ratchet keys.
- prevent double ratchet advancement by not sending messages, delaying break-in recovery.
*cannot:*
- disrupt packet delivery in other queues.
#### An attacker who obtained a connection link
*can:*
- learn the initiating party's chosen router address and public keys.
*cannot:*
- use the link after the intended recipient has completed the connection.
[1]: https://en.wikipedia.org/wiki/End-to-end_encryption
[2]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack
[3]: https://tools.ietf.org/html/rfc5234
+86 -227
View File
@@ -1,4 +1,4 @@
Revision 2, 2024-06-22
Revision 4, 2026-03-09
Evgeny Poberezkin
@@ -8,16 +8,17 @@ Evgeny Poberezkin
- [Introduction](#introduction)
- [What is SimpleX](#what-is-simplex)
- [Network model](#network-model)
- [Applications](#applications)
- [SimpleX objectives](#simplex-objectives)
- [In Comparison](#in-comparison)
- [Technical Details](#technical-details)
- [Trust in Servers](#trust-in-servers)
- [Client -> Server Communication](#client---server-communication)
- [Trust in Routers](#trust-in-routers)
- [Client -> Router Communication](#client---router-communication)
- [2-hop Onion Message Routing](#2-hop-onion-message-routing)
- [SimpleX Messaging Protocol](#simplex-messaging-protocol)
- [SimpleX Agents](#simplex-agents)
- [Encryption Primitives Used](#encryption-primitives-used)
- [Threat model](#threat-model)
- [Security](#security)
- [Acknowledgements](#acknowledgements)
@@ -27,27 +28,27 @@ Evgeny Poberezkin
SimpleX as a whole is a platform upon which applications can be built. [SimpleX Chat](https://github.com/simplex-chat/simplex-chat) is one such application that also serves as an example and reference application.
- [SimpleX Messaging Protocol](./simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a server in-between. The messages are delivered via uni-directional queues created by recipients.
- SMP protocol allows to send message via a SMP server playing proxy role using 2-hop onion routing (referred to as "private routing" in messaging clients) to protect transport information of the sender (IP address and session) from the server chosen (and possibly controlled) by the recipient.
- [SimpleX Messaging Protocol](./simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a router in-between. The messages are delivered via uni-directional queues created by recipients.
- SMP protocol allows to send message via a SMP router playing proxy role using 2-hop onion routing (referred to as "private routing" in messaging clients) to protect transport information of the sender (IP address and session) from the router chosen (and possibly controlled) by the recipient.
- SMP runs over a transport protocol (shown below as TLS) that provides integrity, server authentication, confidentiality, and transport channel binding.
- A SimpleX Server is one of those servers.
- A SimpleX router is one of those routers.
- The SimpleX Network is the term used for the collective of SimpleX Servers that facilitate SMP.
- The SimpleX Network is the term used for the collective of SimpleX routers that facilitate SMP.
- SimpleX Client libraries speak SMP to SimpleX Servers and provide a low-level API not generally intended to be used by applications.
- SimpleX Client libraries speak SMP to SimpleX routers and provide a low-level API not generally intended to be used by applications.
- SimpleX Agents interface with SimpleX Clients to provide a more high-level API intended to be used by applications. Typically they are embedded as libraries, but can also be abstracted into local services.
- SimpleX Agents communicate with other agents inside e2e encrypted envelopes provided by SMP protocol - the syntax and semantics of the messages exchanged by the agent are defined by [SMP agent protocol](./agent-protocol.md)
*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX Server.*
*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX router.*
```
User's Computer Internet Third-Party Server
User's Computer Internet Third-Party Router
------------------ | ---------------------- | -------------------------
| |
SimpleX Chat | |
@@ -57,11 +58,43 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
+----------------+ | |
| SimpleX Agent | | |
+----------------+ -------------- TLS ---------------- +----------------+
| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX Server |
| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX router |
+----------------+ ----------------------------------- +----------------+
| |
```
#### Network model
SimpleX is a general-purpose packet routing network built on top of the Internet. Network endpoints — end-user devices, automated services, AI-enabled applications, IoT devices — exchange data packets through SimpleX network nodes (SMP routers), which accept, buffer, and deliver packets. Each router operates independently and can be operated by any party on standard computing hardware.
SimpleX routers use resource-based addressing: each address identifies a resource on a router, similar to how the World Wide Web addresses resources via URLs. Internet routers, by comparison, use endpoint-based addressing, where IP addresses identify destination devices. Because of this design, SimpleX network participants do not need globally unique addresses to communicate.
SimpleX network has two resource-based addressing schemes:
- *Messaging queues* ([SMP](./simplex-messaging.md)). A queue is a unidirectional, ordered sequence of fixed-size data packets (16,384 bytes each). Each queue has a resource address on a specific router, gated by cryptographic credentials that separately authorize sending and receiving.
- *Data packets* ([XFTP](./xftp.md)). A data packet is an individually addressed block in one of the standard sizes. Each packet has a unique resource address on a specific router, gated by cryptographic credentials. Data packet addressing is more efficient for delivery of larger payloads than queues.
Packet delivery follows a two-router path. The sending endpoint submits a packet to a first router, which forwards it to a second router, where the receiving endpoint retrieves it. The sending endpoint's IP address is known only to the first router; the receiving endpoint's IP address is known only to the second router. See [2-hop Onion Message Routing](#2-hop-onion-message-routing) for details.
Routers buffer packets between submission and retrieval — from seconds to days, enabling asynchronous delivery when endpoints are online at different times. Packets are removed after delivery or after a configured expiration period.
#### Applications
Applications currently using SimpleX network:
- **SimpleX Chat** — a peer-to-peer messenger using SimpleX network as a transport layer, in the same way that communication applications use WebRTC, Tor, i2p, or Nym. All communication logic — contacts, conversations, groups, message formats, end-to-end encryption — runs on endpoint devices.
- **IoT devices** — using the SimpleX queue protocol directly for sensor data collection and device control.
- **AI-based services** — automated services built on the SimpleX Chat application core.
- **Secure monitoring and control systems** — applications for equipment monitoring and control, including robotics, using the network for command delivery and telemetry collection.
[SimpleGo](https://simplego.dev), developed by an independent organization, is a microcontroller-based device running a SimpleX Chat-compatible messenger directly on a microcontroller without a general-purpose operating system. Running over 20 days on a single battery charge, it demonstrates the energy efficiency of resource-based addressing: the device receives packets without continuous polling. A microcontroller-based router implementation that functions simultaneously as a WiFi router is also in development.
#### SimpleX objectives
1. Provide messaging infrastructure for distributed applications. This infrastructure needs to have the following qualities:
@@ -70,7 +103,7 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
- Privacy: protect against traffic correlation attacks to determine the contacts that the users communicate with.
- Reliability: the messages should be delivered even if some participating network servers or receiving clients fail, with at least once delivery guarantee.
- Reliability: the messages should be delivered even if some participating network routers or receiving clients fail, with "at least once" delivery guarantee.
- Integrity: the messages sent in one direction are ordered in a way that sender and recipient agree on; the recipient can detect when a message was removed or changed.
@@ -78,63 +111,63 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX
- Low latency: the delay introduced by the network should not be higher than 100ms-1s in addition to the underlying TCP network latency.
2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious servers.
2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious routers.
3. Balance user experience with privacy requirements, prioritizing experience of mobile device users.
#### In Comparison
SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and servers without depending on any centralized component.
SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and routers without depending on any centralized component.
In comparison to more traditional messaging applications (e.g. WhatsApp, Signal, Telegram) the key differences of SimpleX network are:
- participants do not need to have globally unique addresses to communicate, instead they use redundant unidirectional (simplex) messaging queues, with a separate set of queues for each contact.
- connection requests are passed out-of-band, non-optionally protecting key exchange against man-in-the-middle attack.
- simple message queues provided by network servers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central servers, and content/communication channels.
- simple message queues provided by network routers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central routers, and content/communication channels.
- servers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence.
- routers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence.
- users can change servers with minimal disruption - even after an in-use server disappears, simply by changing the configuration on which servers the new queues are created.
- users can change routers with minimal disruption - even after an in-use router disappears, simply by changing the configuration on which routers the new queues are created.
## Technical Details
#### Trust in Servers
#### Trust in Routers
Clients communicate directly with servers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS.
Clients communicate directly with routers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS.
Users use multiple servers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen servers either directly, if this is a known/trusted server, or via another SMP server providing proxy functionality to protect IP address and session of the sender.
Users use multiple routers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen routers either directly, if this is a known/trusted router, or via another SMP router providing proxy functionality to protect IP address and session of the sender.
Although end-to-end encryption is always present, users place a degree of trust in servers they connect to. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX server is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) servers. While a user *may* re-use a transport connection to fetch messages from multiple queues, or connect to a server from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues.
Although end-to-end encryption is always present, users place a degree of trust in routers they connect to. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX router is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) routers. While a user *may* re-use a transport connection to fetch messages from multiple queues, or connect to a router from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues.
Users may trust a server because:
Users may trust a router because:
- They deploy and control the servers themselves from the available open-source code. This has the trade-offs of strong trust in the server but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that.
- They deploy and control the routers themselves from the available open-source code. This has the trade-offs of strong trust in the router but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that.
- They use servers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers.
- They use routers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers.
By default, servers do not retain access logs, and permanently delete messages and queues when requested. Messages persist only in memory until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a server maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the users communications graph and other meta-data.
By default, routers do not retain access logs, and permanently delete messages and queues when requested. Messages persist in memory or in a database until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a router maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the user's communications graph and other meta-data.
SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in servers. These include rotating the queues in use between users, noise traffic, supporting overlay networks such as Tor, and isolating traffic to different queues to different transport connections (and Tor circuits, if Tor is used).
SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in routers. These include rotating the queues in use between users, noise traffic, supporting overlay networks such as Tor, and isolating traffic to different queues to different transport connections (and Tor circuits, if Tor is used).
[0] While configurable by servers, a minimum value is enforced by the default software. SimpleX Agents can provide redundant routing over queues to mitigate against message loss.
[0] While configurable by routers, a minimum value is enforced by the default software. SimpleX Agents can provide redundant routing over queues to mitigate against message loss.
#### Client -> Server Communication
#### Client -> Router Communication
Utilizing TLS grants the SimpleX Messaging Protocol (SMP) server authentication and metadata protection to a passive network observer. But SMP does not rely on the transport protocol for message confidentiality or client authentication. The SMP protocol itself provides end-to-end confidentiality, authentication, and integrity of messages between communicating parties.
Servers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients, as part of the server address. The offline certificate signs an online certificate used in the transport protocol handshake. [0]
Routers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients, as part of the router address. The offline certificate signs an online certificate used in the transport protocol handshake. [0]
If the transport protocol's confidentiality is broken, incoming and outgoing messages to the server cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the server is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the server can one pass and correlate messages.
If the transport protocol's confidentiality is broken, incoming and outgoing messages to the router cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the router is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the router can one pass and correlate messages.
It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the server incorrectly cannot do anything with the SMP messages except drop them.
It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the router incorrectly cannot do anything with the SMP messages except drop them.
After the connection is established, the client sends blocks of a fixed size 16KB, and the server replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX servers can be deployed as onion services and SimpleX clients can communicate with servers over Tor to further improve participants privacy.
After the connection is established, the client sends blocks of a fixed size 16KB, and the router replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX routers can be deployed as onion services and SimpleX clients can communicate with routers over Tor to further improve participants privacy.
By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a server simultaneously, the server acts as a low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised.
By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a router simultaneously, the router acts as a low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised.
The protocol does not protect against attacks targeted at particular users with known identities - e.g., if the attacker wants to prove that two known users are communicating, they can achieve it by observing their local traffic. At the same time, it substantially complicates large-scale traffic correlation, making determining the real user identities much less effective.
@@ -143,39 +176,39 @@ The protocol does not protect against attacks targeted at particular users with
#### 2-hop Onion Message Routing
As SimpleX Messaging Protocol servers providing messaging queues are chosen by the recipients, in case senders connect to these servers directly the server owners (who potentially can be the recipients themselves) can learn senders' IP addresses (if Tor is not used) and which other queues on the same server are accessed by the user in the same transport connection (even if Tor is used).
As SimpleX Messaging Protocol routers providing messaging queues are chosen by the recipients, in case senders connect to these routers directly the router owners (who potentially can be the recipients themselves) can learn senders' IP addresses (if Tor is not used) and which other queues on the same router are accessed by the user in the same transport connection (even if Tor is used).
While the clients support isolating the messages sent to different queues into different transport connections (and Tor circuits), this is not practical, as it consumes additional traffic and system resources.
To mitigate this problem SimpleX Messaging Protocol servers support 2-hop onion message routing when the SMP server chosen by the sender forwards the messages to the servers chosen by the recipients, thus protecting both the senders IP addresses and sessions, even if connection isolation and Tor are not used.
To mitigate this problem SimpleX Messaging Protocol routers support 2-hop onion message routing when the SMP router chosen by the sender forwards the messages to the routers chosen by the recipients, thus protecting both the senders IP addresses and sessions, even if connection isolation and Tor are not used.
The design of 2-hop onion message routing prevents these potential attacks:
- MITM by proxy (SMP server that forwards the messages).
- MITM by proxy (SMP router that forwards the messages).
- Identification by the proxy which and how many queues the sender sends messages to (as messages are additionally e2e encrypted between the sender and the destination SMP server).
- Identification by the proxy which and how many queues the sender sends messages to (as messages are additionally e2e encrypted between the sender and the destination SMP router).
- Correlation of messages sent to different queues via the same user session (as random correlation IDs and keys are used for each message).
See more details about 2-hop onion message routing design in [SimpleX Messaging Protocol](./simplex-messaging.md#proxying-sender-commands)
Also see [Threat model](#threat-model)
Also see [Security](./security.md)
#### SimpleX Messaging Protocol
SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a server (including IP address or host name, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public keys to agree e2e encryption. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0]
SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a router (including IP address or host name, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public keys to agree e2e encryption. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0]
Because queues are uni-directional, Bob provides an identically-formatted introduction message to Alice over Alice's now-established receiving queue.
When setting up a queue, the server will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the server to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised.
When setting up a queue, the router will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the router to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised.
[0] Users can additionally create public 'contact queues' that are only used to receive connection requests.
[0] Users can additionally create public 'contact queues' that are only used to receive connection requests.
#### SimpleX Agents
SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with servers using SMP. Agent operations include:
SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with routers using SMP. Agent operations include:
- Managing sets of bi-directional, redundant queues for communication partners
@@ -186,195 +219,21 @@ SimpleX agents provide higher-level operations compared to SimpleX Clients, who
- Noise traffic
#### Encryption Primitives Used
## Security
- Ed25519 or Curve25519 to authorize/verify commands to SMP servers (authorization algorithm is set via client/server configuration).
- Curve25519 for DH exchange to agree:
- the shared secret between server and recipient (to encrypt message bodies - it avoids shared cipher-text in sender and recipient traffic)
- the shared secret between sender and recipient (to encrypt messages end-to-end in each queue - it avoids shared cipher-text in redundant queues).
- [NaCl crypto_box](https://nacl.cr.yp.to/box.html) encryption scheme (curve25519xsalsa20poly1305) for message body encryption between server and recipient and for E2E per-queue encryption.
- SHA256 to validate server offline certificates.
- [double ratchet](https://signal.org/docs/specifications/doubleratchet/) protocol for end-to-end message encryption between the agents:
- Curve448 keys to agree shared secrets required for double ratchet initialization (using [X3DH](https://signal.org/docs/specifications/x3dh/) key agreement with 2 ephemeral keys for each side),
- AES-GCM AEAD cipher,
- SHA512-based HKDF for key derivation.
For encryption primitives, threat model, and detailed security analysis, see [Security](./security.md).
SimpleX provides these security properties:
## Threat Model
- **End-to-end encryption** using Double Ratchet algorithm with forward secrecy and post-quantum cryptography.
#### Global Assumptions
- **No shared identifiers** across connections — contacts cannot prove they communicate with the same user.
- A user protects their local database and key material.
- The user's application is authentic, and no local malware is running.
- The cryptographic primitives in use are not broken.
- A user's choice of servers is not directly tied to their identity or otherwise represents distinguishing information about the user.
- The user's client uses 2-hop onion message routing.
- **Sender deniability** — neither routers nor recipients can cryptographically prove message origin.
#### A passive adversary able to monitor the traffic of one user
- **Transport metadata protection** — fixed-size blocks, 2-hop onion routing, and optional connection isolation frustrate traffic correlation.
*can:*
- identify that and when a user is using SimpleX.
- determine which servers the user receives the messages from.
- observe how much traffic is being sent, and make guesses as to its purpose.
*cannot:*
- see who sends messages to the user and who the user sends the messages to.
- determine the servers used by users' contacts.
#### A passive adversary able to monitor a set of senders and recipients
*can:*
- identify who and when is using SimpleX.
- learn which SimpleX Messaging Protocol servers are used as receive queues for which users.
- learn when messages are sent and received.
- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers.
- observe how much traffic is being sent, and make guesses as to its purpose
*cannot, even in case of a compromised transport protocol:*
- perform traffic correlation attacks with any increase in efficiency over a non-compromised transport protocol
#### SimpleX Messaging Protocol server
*can:*
- learn when a queue recipient is online
- know how many messages are sent via the queue (although some may be noise or not content messages).
- learn which messages would trigger notifications even if a user does not use [push notifications](./push-notifications.md).
- perform the correlation of the queue used to receive messages (matching multiple queues to a single user) via either a re-used transport connection, user's IP Address, or connection timing regularities.
- learn a recipient's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used.
- drop all future messages inserted into a queue, detectable only over other, redundant queues.
- lie about the state of a queue to the recipient and/or to the sender (e.g. suspended or deleted when it is not).
- spam a user with invalid messages.
*cannot:*
- undetectably add, duplicate, or corrupt individual messages.
- undetectably drop individual messages, so long as a subsequent message is delivered.
- learn the contents or type of messages.
- distinguish noise messages from regular messages except via timing regularities.
- compromise the users' end-to-end encryption with an active attack.
- learn a sender's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, even if Tor is not used (provided messages are sent via proxy SMP server).
- perform senders' queue correlation (matching multiple queues to a single sender) via either a re-used transport connection, user's IP Address, or connection timing regularities, unless it has additional information from the proxy SMP server (provided messages are sent via proxy SMP server).
#### SimpleX Messaging Protocol server that proxies the messages to another SMP server
*can:*
- learn a sender's IP address, as long as Tor is not used.
- learn when a sender with a given IP address is online.
- know how many messages are sent from a given IP address and to a given destination SMP server.
- drop all messages from a given IP address or to a given destination server.
- unless destination SMP server detects repeated public DH keys of senders, replay messages to a destination server within a single session, causing either duplicate message delivery (which will be detected and ignored by the receiving clients), or, when receiving client is not connected to SMP server, exhausting capacity of destination queues used within the session.
*cannot:*
- perform queue correlation (matching multiple queues to a single user), unless it has additional information from the destination SMP server.
- undetectably add, duplicate, or corrupt individual messages.
- undetectably drop individual messages, so long as a subsequent message is delivered.
- learn the contents or type of messages.
- learn which messages would trigger notifications.
- learn the destination queues of messages.
- distinguish noise messages from regular messages except via timing regularities.
- compromise the user's end-to-end encryption with another user via an active attack.
- compromise the user's end-to-end encryption with the destination SMP servers via an active attack.
#### An attacker who obtained Alice's (decrypted) chat database
*can:*
- see the history of all messages exchanged by Alice with her communication partners.
- see shared profiles of contacts and groups.
- surreptitiously receive new messages sent to Alice via existing queues; until communication queues are rotated or the Double-Ratchet advances forward.
- prevent Alice from receiving all new messages sent to her - either surreptitiously by emptying the queues regularly or overtly by deleting them.
- send messages from the user to their contacts; recipients will detect it as soon as the user sends the next message, because the previous message hash wont match (and potentially wont be able to decrypt them in case they dont keep the previous ratchet keys).
*cannot:*
- impersonate a sender and send messages to the user whose database was stolen. Doing so requires also compromising the server (to place the message in the queue, that is possible until the Double-Ratchet advances forward) or the user's device at a subsequent time (to place the message in the database).
- undetectably communicate at the same time as Alice with her contacts. Doing so would result in the contact getting different messages with repeated IDs.
- undetectably monitor message queues in realtime without alerting the user they are doing so, as a second subscription request unsubscribes the first and notifies the second.
#### A users contact
*can:*
- spam the user with messages.
- forever retain messages from the user.
*cannot:*
- cryptographically prove to a third-party that a message came from a user (assuming the users device is not seized).
- prove that two contacts they have is the same user.
- cannot collaborate with another of the user's contacts to confirm they are communicating with the same user.
#### An attacker who observes Alice showing an introduction message to Bob
*can:*
- Impersonate Bob to Alice.
*cannot:*
- Impersonate Alice to Bob.
#### An attacker with Internet access
*can:*
- Denial of Service SimpleX messaging servers.
- spam a user's public “contact queue” with connection requests.
*cannot:*
- send messages to a user who they are not connected with.
- enumerate queues on a SimpleX server.
- **Out-of-band key exchange** — connection requests passed outside the network protect against MITM attacks.
## Acknowledgements
+82 -5
View File
@@ -13,6 +13,11 @@ Version 1, 2024-06-22
- [Initialization](#initialization)
- [Encrypting messages](#encrypting-messages)
- [Decrypting messages](#decrypting-messages)
- [Ratchet message wire format](#ratchet-message-wire-format)
- [Encrypted ratchet message](#encrypted-ratchet-message)
- [Encrypted message header](#encrypted-message-header)
- [Plaintext message header](#plaintext-message-header)
- [KEM state machine](#kem-state-machine)
- [Implementation considerations](#implementation-considerations)
- [Chosen KEM algorithm](#chosen-kem-algorithm)
- [Summary](#summary)
@@ -71,11 +76,10 @@ def RatchetInitAlicePQ2HE(state, SK, bob_dh_public_key, shared_hka, shared_nhkb,
// below added for post-quantum KEM
state.PQRs = GENERATE_PQKEM()
state.PQRr = bob_pq_kem_encapsulation_key
state.PQRss = random // shared secret for KEM
state.PQRct = PQKEM-ENC(state.PQRr, state.PQRss) // encapsulated additional shared secret
state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext
// above added for KEM
// the next line augments DH key agreement with PQ shared secret
state.RK, state.CKs, state.NHKs = KDF_RK_HE(SK, DH(state.DHRs, state.DHRr) || state.PQRss)
state.RK, state.CKs, state.NHKs = KDF_RK_HE(SK, DH(state.DHRs, state.DHRr) || state.PQRss)
state.CKr = None
state.Ns = 0
state.Nr = 0
@@ -176,8 +180,7 @@ def DHRatchetPQ2HE(state, header):
state.DHRs = GENERATE_DH()
// below is added for KEM
state.PQRs = GENERATE_PQKEM() // generate new PQ key pair
state.PQRss = random // shared secret for KEM
state.PQRct = PQKEM-ENC(state.PQRr, state.PQRss) // encapsulated additional shared secret KEM #1
state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext KEM #1
// above is added for KEM
// use new shared secret with sending ratchet
state.RK, state.CKs, state.NHKs = KDF_RK_HE(state.RK, DH(state.DHRs, state.DHRr) || state.PQRss)
@@ -191,6 +194,80 @@ Other than augmenting DH key agreements with the shared secrets from KEM, the ab
It is worth noting that while DH agreements work as ping-pong, when the new received DH key is used for both DH agreements (and only the sent DH key is updated for the second DH key agreement), PQ KEM agreements in the proposed scheme work as a "parallel ping-pong", with two balls in play all the time (two KEM agreements run in parallel).
## Ratchet message wire format
The pseudocode above describes the algorithm. This section specifies the actual binary encoding used in SimpleX implementation with Curve448 DH keys, sntrup761 KEM and AES-256-GCM AEAD.
The ratchet-encrypted message has three encoding layers, from outermost to innermost:
1. **Encrypted ratchet message** — the complete ratchet message envelope, referenced as an opaque encrypted body in [agent protocol](./agent-protocol.md).
2. **Encrypted message header** — the encrypted header within the ratchet message, used as associated data for message body encryption.
3. **Plaintext message header** — the DH and KEM ratchet keys and counters.
### Encrypted ratchet message
The outer envelope contains the encrypted header (used as associated data for body authentication), the body authentication tag, and the encrypted message body.
The message body is encrypted with AES-256-GCM using the message key derived from the sending chain key (`KDF_CK`). The associated data for body encryption is the concatenation of the ratchet associated data and the encoded encrypted header.
```abnf
encRatchetMessage = versionedLength encMessageHeader msgAuthTag encMsgBody
; encMessageHeader is used as associated data for body decryption: AD = rcAD || encMessageHeader
msgAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the message body
encMsgBody = *OCTET ; AES-256-GCM encrypted padded message body (remaining bytes)
```
### Encrypted message header
The encrypted header wraps the current ratchet e2e encryption version, an initialization vector, an authentication tag, and the encrypted padded header body.
The header body is encrypted with AES-256-GCM using the header key (`HKs`). The associated data for header encryption is the ratchet associated data. The header is padded before encryption to a fixed size to prevent leaking information about the KEM state.
```abnf
encMessageHeader = currentVersion headerIV headerAuthTag versionedLength encHeaderBody
currentVersion = 2*2 OCTET ; Word16, current ratchet e2e encryption version
headerIV = 16*16 OCTET ; AES-256 initialization vector for header encryption
headerAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the header
encHeaderBody = *OCTET ; AES-256-GCM encrypted padded header (see plaintext format below)
```
`versionedLength` uses a 2-byte length prefix (Word16) when the current e2e version supports PQ encryption, or a 1-byte length prefix otherwise. The parser distinguishes the two encodings by peeking at the first byte: values below 32 indicate a 2-byte prefix (as the header is always at least 69 bytes).
```abnf
versionedLength = largeLength / length ; 2-byte for PQ versions, 1-byte for pre-PQ versions
```
The padded header sizes before encryption are: 2310 bytes when PQ is supported, 88 bytes when PQ is not supported. Padding uses a 2-byte big-endian length prefix followed by the plaintext header and `#` fill bytes.
### Plaintext message header
```abnf
msgHeader = maxVersion dhPublicKey [kemParams] prevMsgCount msgCount
maxVersion = 2*2 OCTET ; Word16, max supported e2e encryption version
dhPublicKey = length x509encoded ; Curve448 public DH ratchet key
kemParams = noKEM / proposedKEM / acceptedKEM
; present only when current ratchet version >= pqRatchetE2EEncryptVersion
noKEM = %x30 ; "0" - no KEM parameters
proposedKEM = %x31 %s"P" kemEncapsulationKey ; KEM proposed, not yet accepted
acceptedKEM = %x31 %s"A" kemCiphertext kemEncapsulationKey ; KEM accepted
kemEncapsulationKey = largeLength 1158*1158 OCTET ; sntrup761 encapsulation key
kemCiphertext = largeLength 1039*1039 OCTET ; sntrup761 ciphertext
prevMsgCount = 4*4 OCTET ; Word32, number of messages in previous sending chain
msgCount = 4*4 OCTET ; Word32, message number in current sending chain
length = 1*1 OCTET
largeLength = 2*2 OCTET ; Word16
```
### KEM state machine
PQ encryption can be enabled or disabled during a connection's lifetime. The KEM parameters in the header reflect three states:
- **No KEM** (`noKEM`): PQ encryption is not active. The header contains only the DH key, as in the original double ratchet.
- **Proposed** (`proposedKEM`): One party generated a KEM key pair and includes the encapsulation key in the header, proposing PQ encryption. No ciphertext is included because the other party has not yet sent its encapsulation key.
- **Accepted** (`acceptedKEM`): The party received the other's encapsulation key, performed encapsulation (KEM #1), and includes both the ciphertext and its own new encapsulation key (for KEM #2). This is the steady state for active PQ encryption.
The transition from Proposed to Accepted happens when a party receives a message containing KEM parameters (either Proposed or Accepted) and responds with its own Accepted parameters. Once both parties are in Accepted state, the double PQ KEM augmentation described in the algorithm above operates in each DH ratchet step.
## Implementation considerations for SimpleX Messaging Protocol
As SimpleX Messaging Protocol pads messages to a fixed size, using 16kb transport blocks, the size increase introduced by this scheme can be compensated for by using ZSTD encryption of JSON bodies and image previews encoded as base64. While there may be some rare cases of random texts that would fail to compress, in all real scenarios it would not cause the message size reduction.
+73 -51
View File
@@ -1,14 +1,19 @@
Version 2, 2024-06-22
Version 3, 2025-01-24
# Overview of push notifications for SimpleX Messaging Servers
# Overview of push notifications for SimpleX Messaging Routers
This document describes Notification Router protocol version 3. Version history:
- v1: initial version
- v2: authenticated commands, command batching
- v3: detailed invalid token reason
## Table of contents
- [Introduction](#introduction)
- [Participating servers](#participating-servers)
- [Participating routers](#participating-routers)
- [Register device token to receive push notifications](#register-device-token-to-receive-push-notifications)
- [Subscribe to connection notifications](#subscribe-to-connection-notifications)
- [SimpleX Notification Server protocol](#simplex-notification-server-protocol)
- [SimpleX Notification Router protocol](#simplex-notification-router-protocol)
- [Register new notification token](#register-new-notification-token)
- [Verify notification token](#verify-notification-token)
- [Check notification token status](#check-notification-token-status)
@@ -23,35 +28,35 @@ Version 2, 2024-06-22
## Introduction
SimpleX Messaging servers already operate as push servers and deliver the messages to subscribed clients as soon as they are sent to the servers.
SimpleX Messaging routers already operate as push routers and deliver the messages to subscribed clients as soon as they are sent to the routers.
The reason for push notifications is to support instant message notifications on iOS that does not allow background services.
## Participating servers
## Participating routers
The diagram below shows which servers participate in message notification delivery.
The diagram below shows which routers participate in message notification delivery.
While push provider (e.g., APN) can learn how many notifications are delivered to the user, it cannot access message content, even encrypted, or any message metadata - the notifications are e2e encrypted between SimpleX Notification Server and the user's device.
While push provider (e.g., APN) can learn how many notifications are delivered to the user, it cannot access message content, even encrypted, or any message metadata - the notifications are e2e encrypted between SimpleX Notification Router and the user's device.
```
User's iOS device Internet Servers
User's iOS device Internet Routers
--------------------- . ------------------------ . -----------------------------
. .
. . can be self-hosted now
+--------------+ . . +----------------+
| SimpleX Chat | -------------- TLS --------------- | SimpleX |
| client |------> SimpleX Messaging Protocol (SMP) ------> | Messaging |
+--------------+ ---------------------------------- | Server |
+--------------+ ---------------------------------- | Router |
^ | . . +----------------+
| | . . . . . | . . .
| | . . | V |
| | . . |SMP| TLS
| | . . | | | SimpleX
| | . . . . . V . . . NTF Server
| | . . . . . V . . . NTF Router
| | . . +----------------------------------+
| | . . | +---------------+ |
| | -------------- TLS --------------- | | SimpleX | can be |
| |-----------> Notification Server Protocol -----> | | Notifications | self-hosted |
| |-----------> Notification Router Protocol -----> | | Notifications | self-hosted |
| ---------------------------------- | | Subscriber | in the future |
| . . | +---------------+ |
| . . | | |
@@ -59,7 +64,7 @@ While push provider (e.g., APN) can learn how many notifications are delivered t
| . . | +---------------+ |
| . . | | SimpleX | |
| . . | | Push | |
| . . | | Server | |
| . . | | Router | |
| . . | +---------------+ |
| . . +----------------------------------+
| . . . . . | . . .
@@ -85,25 +90,28 @@ This diagram shows the process of subscription to notifications, notification de
![Subscribe to notifications](./diagrams/notifications/subscription.svg)
## SimpleX Notification Server protocol
## SimpleX Notification Router protocol
To manage notification subscriptions to SMP servers, SimpleX Notification Server provides an RPC protocol with a similar design to SimpleX Messaging Protocol server.
To manage notification subscriptions to SMP routers, SimpleX Notification Router provides an RPC protocol with a similar design to SimpleX Messaging Protocol router.
This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the server certificate is not included in the handshake).
This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the router certificate is not included in the handshake).
The client and router use ALPN extension with `ntf/1` protocol name to agree handshake version.
Protocol commands have this syntax:
```
ntfServerTransmission =
ntfServerCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd /
```abnf
ntfRouterTransmission = authorization corrId entityId ntfRouterCmd
; same transmission structure as SMP, see simplex-messaging.md
ntfRouterCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd /
replaceTokenCmd / deleteTokenCmd / cronCmd /
newSubCmd / checkSubCmd / deleteSubCmd
newSubCmd / checkSubCmd / deleteSubCmd / pingCmd
```
### Register new notification token
This command should be used after the client app obtains a token from push notifications provider to register the token with the server.
This command should be used after the client app obtains a token from push notifications provider to register the token with the router.
Having received this command the server will deliver a test notification via the push provider to validate that the client has this token.
Having received this command the router will deliver a test notification via the push provider to validate that the client has this token.
The command syntax:
@@ -111,23 +119,24 @@ The command syntax:
newTokenCmd = %s"TNEW" SP newToken
newToken = %s"T" deviceToken authPubKey clientDhPubKey
deviceToken = pushProvider tokenString
pushProvider = apnsDev / apnsProd / apnsNull
pushProvider = apnsDev / apnsProd / apnsTest / apnsNull
apnsDev = "AD" ; APNS token for development environment
apnsProd = "AP" ; APNS token for production environment
apnsNull = "AN" ; token that does not trigger any notification delivery - used for server testing
apnsTest = "AT" ; APNS token for test environment (mock server)
apnsNull = "AN" ; token that does not trigger any notification delivery - used for router testing
tokenString = shortString
authPubKey = length x509encoded ; Ed25519 key used to verify clients commands
clientDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the server and client
clientDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client
shortString = length *OCTET
length = 1*1 OCTET
```
The server response syntax:
The router response syntax:
```abnf
tokenIdResp = %s"IDTKN" SP entityId serverDhPubKey
tokenIdResp = %s"IDTKN" SP entityId routerDhPubKey
entityId = shortString
serverDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the server and client
routerDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client
```
### Verify notification token
@@ -159,7 +168,9 @@ The response to this command:
```abnf
tokenStatusResp = %s"TKN" SP tokenStatus
tokenStatus = %s"NEW" / %s"REGISTERED" / %s"INVALID" / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED"
tokenStatus = %s"NEW" / %s"REGISTERED" / tokenInvalid / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED"
tokenInvalid = %s"INVALID" ["," invalidReason] ; optional reason added in v3
invalidReason = %s"BAD" / %s"TOPIC" / %s"EXPIRED" / %s"UNREGISTERED"
```
### Replace notification token
@@ -200,8 +211,8 @@ After this command all message notification subscriptions will be removed and no
This command enables or disables periodic notifications sent to the client device irrespective of message notifications.
This is useful for two reasons:
- it provides better privacy from notification server, as while the server learns the device token, it doesn't learn anything else about user communications.
- it allows to receive messages when notifications were dropped by push provider, e.g. while the device was offline, or lost by notification server, e.g. while it was restarting.
- it provides better privacy from notification router, as while the router learns the device token, it doesn't learn anything else about user communications.
- it allows to receive messages when notifications were dropped by push provider, e.g. while the device was offline, or lost by notification router, e.g. while it was restarting.
The command syntax:
@@ -214,18 +225,18 @@ The interval for periodic notifications is set in minutes, with the minimum of 2
### Create SMP message notification subscription
This command makes notification server subscribe to message notifications from SMP server and to deliver them to push provider:
This command makes notification router subscribe to message notifications from SMP router and to deliver them to push provider:
```abnf
newSubCmd = %s"SNEW" newSub
newSub = %s "S" tokenId smpServer notifierId notifierKey
newSubCmd = %s"SNEW" SP newSub
newSub = %s"S" tokenId smpRouter notifierId notifierKey
tokenId = shortString ; returned in response to `TNEW` command
smpServer = smpServer = hosts port fingerprint
smpRouter = hosts port fingerprint
hosts = length 1*host
host = shortString
port = shortString
fingerprint = shortString
notifierId = shortString ; returned by SMP server in response to `NKEY` SMP command
notifierId = shortString ; returned by SMP router in response to `NKEY` SMP command
notifierKey = length x509encoded ; private key used to authorize requests to subscribe to message notifications
```
@@ -247,10 +258,10 @@ The response:
```abnf
subStatusResp = %s"SUB" SP subStatus
subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP server disconnect/timeout while ntf server is retrying to connect
%s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another server subscribed to notifications
%s"AUTH" / subErrStatus
subErrStatus = %s"ERR" SP shortString
subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP router disconnect/timeout while ntf router is retrying to connect
%s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another router subscribed to notifications
%s"AUTH" / %s"DELETED" / %s"SERVICE" / subErrStatus
subErrStatus = %s"ERR" SP *OCTET
```
### Delete notification subscription
@@ -265,6 +276,17 @@ The response to this command is `okResp` or `errorResp`.
After this command no more message notifications will be sent from this queue.
### Keep-alive command
To keep the transport connection alive the clients should use `PING` command:
```abnf
pingCmd = %s"PING"
pongResp = %s"PONG"
```
This command is sent unsigned and without entity ID.
### Error responses
All commands can return error response:
@@ -277,7 +299,7 @@ Where `errorType` has the same syntax as in [SimpleX Messaging Protocol](./simpl
## Threat Model
This threat model compliments SimpleX Messaging Protocol [threat model](./overview-tjr.md#threat-model)
This threat model compliments SimpleX Messaging Protocol [threat model](./security.md#threat-model)
#### A passive adversary able to monitor the traffic of one user
@@ -287,21 +309,21 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
*cannot:*
- determine which servers a user subscribed to the notifications from.
- determine which routers a user subscribed to the notifications from.
#### A passive adversary able to monitor a set of senders and recipients
*can:*
- perform more efficient traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers.
- perform more efficient traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers.
#### SimpleX Messaging Protocol server
#### SimpleX Messaging Protocol router
*can:*
- learn which messages trigger push notifications.
- learn IP address of SimpleX notification servers used by the user.
- learn IP address of SimpleX notification routers used by the user.
- drop message notifications.
@@ -313,13 +335,13 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
- learn which queues belong to the same users with any additional efficiency compared with not using push notifications.
#### SimpleX Notification Server subscribed to message notifications
#### SimpleX Notification Router subscribed to message notifications
*can:*
- learn a user device token.
- learn how many messaging queues and servers a user receives messages from.
- learn how many messaging queues and routers a user receives messages from.
- learn how many message notifications are delivered to the user from each queue.
@@ -339,7 +361,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
- add, duplicate, or corrupt individual messages that will be shown to the user.
#### SimpleX Notification Server subscribed ONLY to periodic notifications
#### SimpleX Notification Router subscribed ONLY to periodic notifications
*can:*
@@ -351,7 +373,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
*cannot:*
- learn how many messaging queues and servers a user receives messages from.
- learn how many messaging queues and routers a user receives messages from.
- learn how many message notifications are delivered to the user from each queue.
@@ -383,7 +405,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
*cannot:*
- learn which SimpleX Messaging Protocol servers are used by a user (notifications are e2e encrypted).
- learn which SimpleX Messaging Protocol routers are used by a user (notifications are e2e encrypted).
- learn which or how many messaging queues a user receives notifications from.
@@ -395,4 +417,4 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi
- register notification token not present on attacker's device.
- enumerate tokens or subscriptions on a SimpleX Notification Server.
- enumerate tokens or subscriptions on a SimpleX Notification Router.
+215
View File
@@ -0,0 +1,215 @@
Revision 1, 2026-03-09
# SimpleX Network: Security
This document describes the cryptographic primitives and threat model for the SimpleX network. For a general introduction, see [SimpleX: messaging and application platform](./overview-tjr.md).
## Table of contents
- [Encryption primitives](#encryption-primitives)
- [Threat model](#threat-model)
- [Global Assumptions](#global-assumptions)
- [A passive adversary able to monitor the traffic of one user](#a-passive-adversary-able-to-monitor-the-traffic-of-one-user)
- [A passive adversary able to monitor a set of senders and recipients](#a-passive-adversary-able-to-monitor-a-set-of-senders-and-recipients)
- [SimpleX Messaging Protocol router](#simplex-messaging-protocol-router)
- [SimpleX Messaging Protocol router that proxies the messages to another SMP router](#simplex-messaging-protocol-router-that-proxies-the-messages-to-another-smp-router)
- [An attacker who obtained Alice's (decrypted) chat database](#an-attacker-who-obtained-alices-decrypted-chat-database)
- [A user's contact](#a-users-contact)
- [An attacker who observes Alice showing an introduction message to Bob](#an-attacker-who-observes-alice-showing-an-introduction-message-to-bob)
- [An attacker with Internet access](#an-attacker-with-internet-access)
## Encryption primitives
- **Router command authorization**: X25519 DH-based authenticated encryption (SMP v7+), providing sender deniability. Ed25519 signatures used for recipient commands and notifier commands.
- **Per-queue key agreement**: Curve25519 DH exchange to agree:
- the shared secret between router and recipient (to encrypt message bodies — avoids shared ciphertext in sender and recipient traffic),
- the shared secret between sender and recipient (to encrypt messages end-to-end in each queue — avoids shared ciphertext in redundant queues).
- **SMP-layer encryption**: [NaCl crypto_box](https://nacl.cr.yp.to/box.html) (curve25519xsalsa20poly1305) for message body encryption between router and recipient, and for e2e per-queue encryption.
- **Certificate validation**: SHA256 to validate router offline certificates.
- **End-to-end encryption**: [Double ratchet](https://signal.org/docs/specifications/doubleratchet/) protocol:
- Curve448 keys for shared secret agreement via [X3DH](https://signal.org/docs/specifications/x3dh/) with 2 ephemeral keys per side,
- optional [SNTRUP761](https://ntruprime.cr.yp.to/) post-quantum KEM running in parallel with the DH ratchet (see [PQDR](./pqdr.md)), providing post-quantum forward secrecy,
- AES-GCM AEAD cipher,
- SHA512-based HKDF for key derivation.
## Threat Model
### Global Assumptions
- A user protects their local database and key material.
- The user's application is authentic, and no local malware is running.
- The cryptographic primitives in use are not broken.
- A user's choice of routers is not directly tied to their identity or otherwise represents distinguishing information about the user.
- The user's client uses 2-hop onion message routing.
### A passive adversary able to monitor the traffic of one user
*can:*
- identify that and when a user is using SimpleX.
- determine which routers the user receives messages from.
- observe how much traffic is being sent, and make guesses as to its purpose.
*cannot:*
- see who sends messages to the user and who the user sends messages to.
- determine the routers used by users' contacts.
### A passive adversary able to monitor a set of senders and recipients
*can:*
- identify who and when is using SimpleX.
- learn which SimpleX Messaging Protocol routers are used as receive queues for which users.
- learn when messages are sent and received.
- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers.
- observe how much traffic is being sent, and make guesses as to its purpose.
*cannot, even in case of a compromised transport protocol:*
- perform traffic correlation attacks with any increase in efficiency over a non-compromised transport protocol.
### SimpleX Messaging Protocol router
*can:*
- learn when a queue recipient is online.
- know how many messages are sent via the queue (although some may be noise or not content messages).
- learn which messages would trigger notifications even if a user does not use [push notifications](./push-notifications.md).
- perform the correlation of the queue used to receive messages (matching multiple queues to a single user) via either a re-used transport connection, user's IP Address, or connection timing regularities.
- learn a recipient's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used.
- drop all future messages inserted into a queue, detectable only over other, redundant queues.
- lie about the state of a queue to the recipient and/or to the sender (e.g. suspended or deleted when it is not).
- spam a user with invalid messages.
*cannot:*
- undetectably add, duplicate, or corrupt individual messages.
- undetectably drop individual messages, so long as a subsequent message is delivered.
- learn the contents or type of messages.
- distinguish noise messages from regular messages except via timing regularities.
- compromise the users' end-to-end encryption with an active attack.
- learn a sender's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, even if Tor is not used (provided messages are sent via proxy SMP router).
- perform senders' queue correlation (matching multiple queues to a single sender) via either a re-used transport connection, user's IP Address, or connection timing regularities, unless it has additional information from the proxy SMP router (provided messages are sent via proxy SMP router).
### SimpleX Messaging Protocol router that proxies the messages to another SMP router
*can:*
- learn a sender's IP address, as long as Tor is not used.
- learn when a sender with a given IP address is online.
- know how many messages are sent from a given IP address and to a given destination SMP router.
- drop all messages from a given IP address or to a given destination router.
- unless destination SMP router detects repeated public DH keys of senders, replay messages to a destination router within a single session, causing either duplicate message delivery (which will be detected and ignored by the receiving clients), or, when receiving client is not connected to SMP router, exhausting capacity of destination queues used within the session.
*cannot:*
- perform queue correlation (matching multiple queues to a single user), unless it has additional information from the destination SMP router.
- undetectably add, duplicate, or corrupt individual messages.
- undetectably drop individual messages, so long as a subsequent message is delivered.
- learn the contents or type of messages.
- learn which messages would trigger notifications.
- learn the destination queues of messages.
- distinguish noise messages from regular messages except via timing regularities.
- compromise the user's end-to-end encryption with another user via an active attack.
- compromise the user's end-to-end encryption with the destination SMP routers via an active attack.
### An attacker who obtained Alice's (decrypted) chat database
*can:*
- see the history of all messages exchanged by Alice with her communication partners.
- see shared profiles of contacts and groups.
- surreptitiously receive new messages sent to Alice via existing queues; until communication queues are rotated or the Double-Ratchet advances forward.
- prevent Alice from receiving all new messages sent to her - either surreptitiously by emptying the queues regularly or overtly by deleting them.
- send messages from the user to their contacts; recipients will detect it as soon as the user sends the next message, because the previous message hash won't match (and potentially won't be able to decrypt them in case they don't keep the previous ratchet keys).
*cannot:*
- impersonate a sender and send messages to the user whose database was stolen. Doing so requires also compromising the router (to place the message in the queue, that is possible until the Double-Ratchet advances forward) or the user's device at a subsequent time (to place the message in the database).
- undetectably communicate at the same time as Alice with her contacts. Doing so would result in the contact getting different messages with repeated IDs.
- undetectably monitor message queues in realtime without alerting the user they are doing so, as a second subscription request unsubscribes the first and notifies the first.
### A user's contact
*can:*
- spam the user with messages.
- forever retain messages from the user.
*cannot:*
- cryptographically prove to a third-party that a message came from a user (assuming the user's device is not seized).
- prove that two contacts they have is the same user.
- cannot collaborate with another of the user's contacts to confirm they are communicating with the same user.
### An attacker who observes Alice showing an introduction message to Bob
*can:*
- Impersonate Bob to Alice.
*cannot:*
- Impersonate Alice to Bob.
### An attacker with Internet access
*can:*
- Denial of Service SimpleX messaging routers.
- spam a user's public "contact queue" with connection requests.
*cannot:*
- send messages to a user who they are not connected with.
- enumerate queues on a SimpleX router.
File diff suppressed because it is too large Load Diff
+199 -157
View File
@@ -1,4 +1,4 @@
Version 2, 2024-06-22
Version 3, 2025-01-24
# SimpleX File Transfer Protocol
@@ -11,12 +11,12 @@ Version 2, 2024-06-22
- [XFTP procedure](#xftp-procedure)
- [File description](#file-description)
- [URIs syntax](#uris-syntax)
- [XFTP server URI](#xftp-server-uri)
- [XFTP router URI](#xftp-router-uri)
- [File description URI](#file-description-URI)
- [XFTP qualities and features](#xftp-qualities-and-features)
- [Cryptographic algorithms](#cryptographic-algorithms)
- [File chunk IDs](#file-chunk-ids)
- [Server security requirements](#server-security-requirements)
- [Data packet IDs](#data-packet-ids)
- [Router security requirements](#router-security-requirements)
- [Transport protocol](#transport-protocol)
- [TLS ALPN](#tls-alpn)
- [Connection handshake](#connection-handshake)
@@ -26,13 +26,14 @@ Version 2, 2024-06-22
- [Command authentication](#command-authentication)
- [Keep-alive command](#keep-alive-command)
- [File sender commands](#file-sender-commands)
- [Register new file chunk](#register-new-file-chunk)
- [Add file chunk recipients](#add-file-chunk-recipients)
- [Upload file chunk](#upload-file-chunk)
- [Delete file chunk](#delete-file-chunk)
- [Register new data packet](#register-new-data-packet)
- [Add data packet recipients](#add-data-packet-recipients)
- [Upload data packet](#upload-data-packet)
- [Delete data packet](#delete-data-packet)
- [File recipient commands](#file-recipient-commands)
- [Download file chunk](#download-file-chunk)
- [Acknowledge file chunk download](#acknowledge-file-chunk-download)
- [Download data packet](#download-data-packet)
- [Acknowledge data packet download](#acknowledge-data-packet-download)
- [Error responses](#error-responses)
- [Threat model](#threat-model)
## Abstract
@@ -45,23 +46,31 @@ It is designed as a application level protocol to solve the problem of secure an
## Introduction
The objective of SimpleX File Transfer Protocol (XFTP) is to facilitate the secure and private unidirectional transfer of files from senders to recipients via persistent file chunks stored by the xftp server.
The objective of SimpleX File Transfer Protocol (XFTP) is to facilitate the secure and private unidirectional transfer of files from senders to recipients via persistent data packets stored by the xftp router.
XFTP is implemented as an application level protocol on top of HTTP2 and TLS.
The protocol describes the set of commands that senders and recipients can send to XFTP servers to create, upload, download and delete file chunks of several pre-defined sizes. XFTP servers SHOULD support chunks of 4 sizes: 64KB, 256KB, 1MB and 4MB (1KB = 1024 bytes, 1MB = 1024KB).
This document describes XFTP protocol version 3. The version history:
The protocol is designed with the focus on meta-data privacy and security. While using TLS, the protocol does not rely on TLS security by using additional encryption to achieve that there are no identifiers or ciphertext in common in received and sent server traffic, frustrating traffic correlation even if TLS is compromised.
- v1: initial version
- v2: authenticated commands - added basic auth support for commands
- v3: blocked files - added BLOCKED error type for policy violations
XFTP does not use any form of participants' identities. It relies on out-of-band passing of "file description" - a human-readable YAML document with the list of file chunk locations, hashes and necessary cryptographic keys.
The protocol describes the set of commands that senders and recipients can send to XFTP routers to create, upload, download and delete data packets of several pre-defined sizes. XFTP routers SHOULD support packets of 4 sizes: 64KB, 256KB, 1MB and 4MB (1KB = 1024 bytes, 1MB = 1024KB).
The protocol is designed with the focus on meta-data privacy and security. While using TLS, the protocol does not rely on TLS security by using additional encryption to achieve that there are no identifiers or ciphertext in common in received and sent router traffic, frustrating traffic correlation even if TLS is compromised.
XFTP does not use any form of participants' identities. It relies on out-of-band passing of "file description" - a human-readable YAML document with the list of data packet locations, hashes and necessary cryptographic keys.
> **Note:** While this protocol was originally designed for file transfer, it handles generic addressed data packets. File-specific semantics (splitting files into packets, assembly, naming) are application-level concerns defined in the [agent protocol](./agent-protocol.md).
## XFTP Model
The XFTP model has three communication participants: the recipient, the file server (XFTP server) that is chosen and, possibly, controlled by the sender, and the sender.
The XFTP model has three communication participants: the recipient, the XFTP router that is chosen and, possibly, controlled by the sender, and the sender.
XFTP server allows uploading fixed size file chunks, with or without basic authentication. The same party that can be the sender of one file chunk can be the recipient of another, without exposing it to the server.
XFTP router allows uploading fixed size data packets, with or without basic authentication. The same party that can be the sender of one data packet can be the recipient of another, without exposing it to the router.
Each file chunk allows multiple recipients, each recipient can download the same chunk multiple times. It allows depending on the threat model use the same recipient credentials for multiple parties, thus reducing server ability to understand the number of intended recipients (but server can still track IP addresses to determine it), or use one unique set of credentials for each recipient, frustrating traffic correlation on the assumption of compromised TLS. In the latter case, senders can create a larger number of recipient credentials to hide the actual number of intended recipients from the servers (which is what SimpleX clients do).
Each data packet allows multiple recipients, each recipient can download the same packet multiple times. It allows depending on the threat model use the same recipient credentials for multiple parties, thus reducing router ability to understand the number of intended recipients (but router can still track IP addresses to determine it), or use one unique set of credentials for each recipient, frustrating traffic correlation on the assumption of compromised TLS. In the latter case, senders can create a larger number of recipient credentials to hide the actual number of intended recipients from the routers (which is what SimpleX clients do).
```
Sender Internet XFTP relays Internet Recipient
@@ -69,7 +78,7 @@ Each file chunk allows multiple recipients, each recipient can download the same
| | | |
| | (can be self-hosted) | |
| | +---------+ | |
chunk 1 ----- HTTP2 over TLS ------ | XFTP | ---- HTTP2 / TLS ----- chunk 1
packet 1 ----- HTTP2 over TLS ------ | XFTP | ---- HTTP2 / TLS ----- packet 1
|---> SimpleX File Transfer Protocol (XFTP) --> | Relay | ---> XFTP ------------->|
| --------------------------- +---------+ ---------------------- |
| | | | | |
@@ -83,21 +92,21 @@ file ---> | XFTP | ------> XFTP ----> | Relay | --->
| | | +---------+ | | |
| ------- HTTP2 / TLS ------- | XFTP | ---- HTTP2 / TLS ---- |
|-------------> XFTP ----> | Relay | ---> XFTP ------------->|
chunk N --------------------------- +---------+ --------------------- chunk N
| | (store file chunks) | |
packet N --------------------------- +---------+ --------------------- packet N
| | (store data packets) | |
| | | |
| | | |
```
When sender client uploads a file chunk, it has to register it first with one sender ID and multiple recipient IDs, and one random unique key per ID to authenticate sender and recipients, and also provide its size and hash that will be validated when chunk is uploaded.
When sender client uploads a data packet, it has to register it first with one sender ID and multiple recipient IDs, and one random unique key per ID to authenticate sender and recipients, and also provide its size and hash that will be validated when packet is uploaded.
To send the actual file, the sender client MUST pad it and encrypt it with a random symmetric key and distribute chunks of fixed sized across multiple XFTP servers. Information about chunk locations, keys, hashes and required keys is passed to the recipients as "[file description](#file-description)" out-of-band.
To send the actual file, the sender client MUST pad it and encrypt it with a random symmetric key and distribute packets of fixed sized across multiple XFTP routers. Information about packet locations, keys, hashes and required keys is passed to the recipients as "[file description](#file-description)" out-of-band.
Creating, uploading, downloading and deleting file chunks requires sending commands to the XFTP server - they are described in detail in [XFTP commands](#xftp-commands) section.
Creating, uploading, downloading and deleting data packets requires sending commands to the XFTP router - they are described in detail in [XFTP commands](#xftp-commands) section.
## Persistence model
Server stores file chunk records in memory, with optional adding to append-only log, to allow restoring them on server restart. File chunk bodies can be stored as files or as objects in any object store (e.g. S3).
Router stores data packet records in memory, with optional adding to append-only log, to allow restoring them on router restart. Data packet bodies can be stored as files or as objects in any object store (e.g. S3).
## XFTP procedure
@@ -107,28 +116,28 @@ To send the file, the sender will:
1) Prepare file
- compute its SHA512 digest.
- prepend header with the name and pad the file to match the whole number of chunks in size. It is RECOMMENDED to use 2 of 4 allowed chunk sizes, to balance upload size and metadata privacy.
- prepend header with the name and pad the file to match the whole number of packets in size. It is RECOMMENDED to use 2 of 4 allowed packet sizes, to balance upload size and metadata privacy.
- encrypt it with a randomly chosen symmetric key and IV (e.g., using NaCL secret_box).
- split into allowed size chunks.
- split into allowed size packets.
- generate per-recipient keys. It is recommended that the sending client generates more per-recipient keys than the actual number of recipients, rounding up to a power of 2, to conceal the actual number of intended recipients.
2) Upload file chunks
- register each chunk record with randomly chosen one or more (for redundancy) XFTP server(s).
2) Upload data packets
- register each packet record with randomly chosen one or more (for redundancy) XFTP router(s).
- optionally request additional recipient IDs, if required number of recipient keys didn't fit into register request.
- upload each chunk to chosen server(s).
- upload each packet to chosen router(s).
3) Prepare file descriptions, one per recipient.
The sending client combines addresses of all chunks and other information into "file description", different for each file recipient, that will include:
The sending client combines addresses of all packets and other information into "file description", different for each file recipient, that will include:
- an encryption key used to encrypt/decrypt the full file (the same for all recipients).
- file SHA512 digest to validate download.
- list of chunk descriptions; information for each chunk:
- private Ed25519 key to sign commands for file transfer server.
- chunk address (server host and chunk ID).
- chunk sha512 digest.
- list of packet descriptions; information for each packet:
- private Ed25519 key to sign commands for file transfer router.
- packet address (router host and packet ID).
- packet sha256 digest.
To reduce the size of file description, chunks are grouped by the server host.
To reduce the size of file description, packets are grouped by the router host.
4) Send file description(s) to the recipient(s) out-of-band, via pre-existing secure and authenticated channel. E.g., SimpleX clients send it as messages via SMP protocol, but it can be done via any other channel.
@@ -138,16 +147,16 @@ To reduce the size of file description, chunks are grouped by the server host.
Having received the description, the recipient will:
1) Download all chunks.
1) Download all packets.
The receiving client can fall back to secondary servers, if necessary:
- if the server is not available.
- if the chunk is not present on the server (ERR AUTH response).
- if the hash of the downloaded file chunk does not match the description.
The receiving client can fall back to secondary routers, if necessary:
- if the router is not available.
- if the packet is not present on the router (ERR AUTH response).
- if the hash of the downloaded data packet does not match the description.
Optionally recipient can acknowledge file chunk reception to delete file ID from server for this recipient.
Optionally recipient can acknowledge data packet reception to delete file ID from router for this recipient.
2) Combine the chunks into a file.
2) Combine the packets into a file.
3) Decrypt the file using the key in file description.
@@ -163,35 +172,35 @@ Optionally recipient can acknowledge file chunk reception to delete file ID from
It includes these fields:
- `party` - "sender" or "recipient". Sender's file description is required to delete the file.
- `size` - padded file size equal to total size of all chunks, see `fileSize` syntax below.
- `size` - padded file size equal to total size of all packets, see `fileSize` syntax below.
- `digest` - SHA512 hash of encrypted file, base64url encoded string.
- `key` - symmetric encryption key to decrypt the file, base64url encoded string.
- `nonce` - nonce to decrypt the file, base64url encoded string.
- `chunkSize` - default chunk size, see `fileSize` syntax below.
- `replicas` - the array of file chunk replicas descriptions.
- `chunkSize` - default packet size, see `fileSize` syntax below.
- `replicas` - the array of data packet replicas descriptions.
- `redirect` - optional property for redirect information indicating that the file is itself a description to another file, allowing to use file description as a short URI.
Each replica description is an object with 2 fields:
- `chunks` - and array of chunk replica descriptions stored on one server.
- `server` - [server address](#xftp-server-uri) where the chunks can be downloaded from.
- `chunks` - an array of packet replica descriptions stored on one server.
- `server` - [router address](#xftp-router-uri) where the packets can be downloaded from.
Each server replica description is a string with this syntax:
Each router replica description is a string with this syntax:
```abnf
chunkReplica = chunkNo ":" replicaId ":" replicaKey [":" chunkDigest [":" chunkSize]]
chunkNo = 1*DIGIT
; a sequential 1-based chunk number in the original file.
packetReplica = packetNo ":" replicaId ":" replicaKey [":" packetDigest [":" packetSize]]
packetNo = 1*DIGIT
; a sequential 1-based packet number in the original file.
replicaId = base64url
; server-assigned random chunk replica ID.
; router-assigned random packet replica ID.
replicaKey = base64url
; sender-generated random key to receive (or to delete, in case of sender's file description) the chunk replica.
chunkDigest = base64url
; chunk digest that MUST be specified for the first replica of each chunk,
; sender-generated random key to receive (or to delete, in case of sender's file description) the packet replica.
packetDigest = base64url
; packet digest that MUST be specified for the first replica of each packet,
; and SHOULD be omitted (or be the same) on the subsequent replicas
chunkSize = fileSize
packetSize = fileSize
fileSize = sizeInBytes / sizeInUnits
; chunk size SHOULD only be specified on the first replica and only if it is different from default chunk size
; packet size SHOULD only be specified on the first replica and only if it is different from default packet size
sizeInBytes = 1*DIGIT
sizeInUnits = 1*DIGIT sizeUnit
sizeUnit = %s"kb" / %s"mb" / %s"gb"
@@ -204,28 +213,28 @@ Optional redirect information has two fields:
## URIs syntax
### XFTP server URI
### XFTP router URI
The XFTP server address is a URI with the following syntax:
The XFTP router address is a URI with the following syntax:
```abnf
xftpServerURI = %s"xftp://" xftpServer
xftpServer = serverIdentity [":" basicAuth] "@" srvHost [":" port]
xftpRouterURI = %s"xftp://" xftpRouter
xftpRouter = routerIdentity [":" basicAuth] "@" srvHost [":" port]
srvHost = <hostname> ; RFC1123, RFC5891
port = 1*DIGIT
serverIdentity = base64url
routerIdentity = base64url
basicAuth = base64url
```
### File description URI
This file description URI can be generated by the client application to share a small file description as a QR code or as a link. Practically, to be able to scan a QR code it should be under 1000 characters, so only file descriptions with 1-2 chunks can be used in this case. This is supported with `redirect` property when file description leads to a file which in itself is a larger file description to another file - akin to URL shortener.
This file description URI can be generated by the client application to share a small file description as a QR code or as a link. Practically, to be able to scan a QR code it should be under 1000 characters, so only file descriptions with 1-2 packets can be used in this case. This is supported with `redirect` property when file description leads to a file which in itself is a larger file description to another file - akin to URL shortener.
File description URI syntax:
```abnf
fileDescriptionURI = serviceScheme "/file" "#/?desc=" description [ "&data=" userData ]
serviceScheme = (%s"https://" clientAppServer) | %s"simplex:"
serviceScheme = (%s"https://" clientAppServer) / %s"simplex:"
clientAppServer = hostname [ ":" port ]
; client app server, e.g. simplex.chat
description = <URI-escaped YAML file description>
@@ -240,50 +249,50 @@ clientAppServer is not a server the client connects to - it is a server that sho
XFTP stands for SimpleX File Transfer Protocol. Its design is based on the same ideas and has some of the qualities of SimpleX Messaging Protocol:
- recipient cannot see sender's IP address, as the file fragments (chunks) are temporarily stored on multiple XFTP relays.
- recipient cannot see sender's IP address, as the file fragments (packets) are temporarily stored on multiple XFTP relays.
- file can be sent asynchronously, without requiring the sender to be online for file to be received.
- there is no network of peers that can observe this transfer - sender chooses which XFTP relays to use, and can self-host their own.
- XFTP relays do not have any file metadata - they only see individual chunks, with access to each chunk authorized with anonymous credentials (using Edwards curve cryptographic signature) that are random per chunk.
- chunks have one of the sizes allowed by the servers - 64KB, 256KB, 1MB and 4MB chunks, so sending a large file looks indistinguishable from sending many small files to XFTP server. If the same transport connection is reused, server would only know that chunks are sent by the same user.
- each chunk can be downloaded by multiple recipients, but each recipient uses their own key and chunk ID to authorize access, and the chunk is encrypted by a different key agreed via ephemeral DH keys (NaCl crypto_box (SalsaX20Poly1305 authenticated encryption scheme ) with shared secret derived from Curve25519 key exchange) on the way from the server to each recipient. XFTP protocol as a result has the same quality as SMP protocol - there are no identifiers and ciphertext in common between sent and received traffic inside TLS connection, so even if TLS is compromised, it complicates traffic correlation attacks.
- XFTP protocol supports redundancy - each file chunk can be sent via multiple relays, and the recipient can choose the one that is available. Current implementation of XFTP protocol in SimpleX Chat does not support redundancy though.
- XFTP relays do not have any file metadata - they only see individual packets, with access to each packet authorized with anonymous credentials (using Edwards curve cryptographic signature) that are random per packet.
- packets have one of the sizes allowed by the routers - 64KB, 256KB, 1MB and 4MB packets, so sending a large file looks indistinguishable from sending many small files to XFTP router. If the same transport connection is reused, router would only know that packets are sent by the same user.
- each packet can be downloaded by multiple recipients, but each recipient uses their own key and packet ID to authorize access, and the packet is encrypted by a different key agreed via ephemeral DH keys (NaCl crypto_box (SalsaX20Poly1305 authenticated encryption scheme ) with shared secret derived from Curve25519 key exchange) on the way from the router to each recipient. XFTP protocol as a result has the same quality as SMP protocol - there are no identifiers and ciphertext in common between sent and received traffic inside TLS connection, so even if TLS is compromised, it complicates traffic correlation attacks.
- XFTP protocol supports redundancy - each data packet can be sent via multiple relays, and the recipient can choose the one that is available. Current implementation of XFTP protocol in SimpleX Chat does not support redundancy though.
- the file as a whole is encrypted with a random symmetric key using NaCl secret_box.
## Cryptographic algorithms
Clients must cryptographically authorize XFTP commands, see [Command authentication](#command-authentication).
To authorize/verify transmissions clients and servers MUST use either signature algorithm Ed25519 algorithm defined in RFC8709 or using deniable authentication scheme based on NaCL crypto_box (see Simplex Messaging Protocol).
To authorize/verify transmissions clients and routers MUST use either signature algorithm Ed25519 algorithm defined in RFC8709 or using deniable authentication scheme based on NaCL crypto_box (see Simplex Messaging Protocol).
To encrypt/decrypt file chunk bodies delivered to the recipients, servers/clients MUST use NaCL crypto_box.
To encrypt/decrypt data packet bodies delivered to the recipients, routers/clients MUST use NaCL crypto_box.
Clients MUST encrypt file chunk bodies sent via XFTP servers using use NaCL crypto_box.
Clients MUST encrypt data packet bodies sent via XFTP routers using use NaCL crypto_box.
## File chunk IDs
## Data packet IDs
XFTP servers MUST generate a separate new set of IDs for each new chunk - for the sender (that uploads the chunk) and for each intended recipient. It is REQUIRED that:
XFTP routers MUST generate a separate new set of IDs for each new packet - for the sender (that uploads the packet) and for each intended recipient. It is REQUIRED that:
- These IDs are different and unique within the server.
- These IDs are different and unique within the router.
- Based on random bytes generated with cryptographically strong pseudo-random number generator.
## Server security requirements
## Router security requirements
XFTP server implementations MUST NOT create, store or send to any other servers:
XFTP router implementations MUST NOT create, store or send to any other routers:
- Logs of the client commands and transport connections in the production environment.
- History of retrieved files.
- Snapshots of the database they use to store file chunks (instead clients can manage redundancy by creating chunk replicas using more than one XFTP server). In-memory persistence is recommended for file chunks records.
- Snapshots of the database they use to store data packets (instead clients can manage redundancy by creating packet replicas using more than one XFTP router). In-memory persistence is recommended for data packets records.
- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using XFTP servers.
- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using XFTP routers.
## Transport protocol
- binary-encoded commands sent as fixed-size padded block in the body of HTTP2 POST request, similar to SMP and notifications server protocol transmission encodings.
- binary-encoded commands sent as fixed-size padded block in the body of HTTP2 POST request, similar to SMP and notifications router protocol transmission encodings.
- HTTP2 POST with a fixed size padded block body for file upload and download.
Block size - 4096 bytes (it would fit ~120 Ed25519 recipient keys).
Block size - 16384 bytes (it would fit ~350 Ed25519 recipient keys).
The reasons to use HTTP2:
@@ -299,40 +308,41 @@ The reason not to use URI segments / HTTP verbs / REST semantics is to have cons
### ALPN to agree handshake version
Client and server use [ALPN extension][18] of TLS to agree handshake version.
Client and router use [ALPN extension][18] of TLS to agree handshake version.
Server SHOULD send `xftp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax.
Router SHOULD send `xftp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax.
If the client does not confirm this protocol name, the server would fall back to v1 of XFTP protocol.
If the client does not confirm this protocol name, the router would fall back to v1 of XFTP protocol.
### Transport handshake
When a client and a server agree on handshake version using ALPN extension, they should proceed with XFTP handshake.
When a client and a router agree on handshake version using ALPN extension, they should proceed with XFTP handshake.
As with SMP, a client doesn't reveal its version range to avoid version fingerprinting. Unlike SMP, XFTP runs a HTTP2 protocol over TLS and the server can't just send its handshake right away. So a session handshake is driven by client-sent requests:
As with SMP, a client doesn't reveal its version range to avoid version fingerprinting. Unlike SMP, XFTP runs a HTTP2 protocol over TLS and the router can't just send its handshake right away. So a session handshake is driven by client-sent requests:
1. To pass initiative to the server, the client sends a request with empty body.
2. Server responds with its `paddedServerHello` block.
1. To pass initiative to the router, the client sends a request with empty body.
2. Router responds with its `paddedRouterHello` block.
3. Clients sends a request containing `paddedClientHello` block,
4. Server sends an empty response, finalizing the handshake.
4. Router sends an empty response, finalizing the handshake.
Once TLS handshake is complete, client and server will exchange blocks of fixed size (16384 bytes).
Once TLS handshake is complete, client and router will exchange blocks of fixed size (16384 bytes).
```abnf
paddedServerHello = <padded(serverHello, 16384)>
serverHello = xftpVersionRange sessionIdentifier serverCert signedServerKey ignoredPart
paddedRouterHello = <padded(routerHello, 16384)>
routerHello = xftpVersionRange sessionIdentifier routerCerts signedRouterKey ignoredPart
xftpVersionRange = minXftpVersion maxXftpVersion
minXftpVersion = xftpVersion
maxXftpVersion = xftpVersion
sessionIdentifier = shortString
; unique session identifier derived from transport connection handshake
serverCert = originalLength <x509encoded>
signedServerKey = originalLength <x509encoded> ; signed by server certificate
routerCerts = length 1*routerCert ; NonEmpty list of certificates in chain
routerCert = originalLength <x509encoded>
signedRouterKey = originalLength <x509encoded> ; signed by router certificate
paddedClientHello = <padded(clientHello, 16384)>
clientHello = xftpVersion keyHash ignoredPart
; chosen XFTP protocol version - must be the maximum supported version
; within the range offered by the server
; within the range offered by the router
xftpVersion = 2*2OCTET ; Word16 version number
keyHash = shortString
@@ -342,47 +352,47 @@ originalLength = 2*2OCTET
ignoredPart = *OCTET
```
In XFTP v2 the handshake is only used for version negotiation, but `serverCert` and `signedServerKey` must be validated by the client.
In XFTP v2 the handshake is only used for version negotiation, but `routerCert` and `signedRouterKey` must be validated by the client.
`keyHash` is the CA fingerprint used by client to validate TLS certificate chain and is checked by a server against its own key.
`keyHash` is the CA fingerprint used by client to validate TLS certificate chain and is checked by a router against its own key.
`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and servers must ignore any extra bytes within the original block length.
`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and routers must ignore any extra bytes within the original block length.
For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `serverHello` block to allow communication over some other transport protocol (possibly, with another channel binding).
For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `routerHello` block to allow communication over some other transport protocol (possibly, with another channel binding).
### Requests and responses
- File sender:
- create file chunk record.
- create data packet record.
- Parameters:
- Ed25519 key for subsequent sender commands and Ed25519 keys for commands of each recipient.
- chunk size.
- packet size.
- Response:
- chunk ID for the sender and different IDs for all recipients.
- add recipients to file chunk
- packet ID for the sender and different IDs for all recipients.
- add recipients to data packet
- Parameters:
- sender's chunk ID
- sender's packet ID
- Ed25519 keys for commands of each recipient.
- Response:
- chunk IDs for new recipients.
- upload file chunk.
- delete file chunk (invalidates all recipient IDs).
- packet IDs for new recipients.
- upload data packet.
- delete data packet (invalidates all recipient IDs).
- File recipient:
- download file chunk:
- chunk ID
- DH key for additional encryption of the chunk.
- command should be signed with the key passed by the sender when creating chunk record.
- delete file chunk ID (only for one recipient): signed with the same key.
- download data packet:
- packet ID
- DH key for additional encryption of the packet.
- command should be signed with the key passed by the sender when creating packet record.
- delete data packet ID (only for one recipient): signed with the same key.
## XFTP commands
Commands syntax below is provided using ABNF with case-sensitive strings extension.
```abnf
xftpCommand = ping / senderCommand / recipientCmd / serverMsg
xftpCommand = ping / senderCommand / recipientCmd / routerMsg
senderCommand = register / add / put / delete
recipientCmd = get / ack
serverMsg = pong / sndIds / rcvIds / ok / file
routerMsg = pong / sndIds / rcvIds / ok / file / error
```
The syntax of specific commands and responses is defined below.
@@ -393,11 +403,11 @@ Commands are made via HTTP2 requests, responses to commands are correlated as HT
### Command authentication
XFTP servers must authenticate all transmissions (excluding `ping`) by verifying the client signatures. Command signature should be generated by applying the algorithm specified for the file to the `signed` block of the transmission, using the key associated with the file chunk ID (recipient's or sender's depending on which file chunk ID is used).
XFTP routers must authenticate all transmissions (excluding `ping`) by verifying the client signatures. Command signature should be generated by applying the algorithm specified for the file to the `signed` block of the transmission, using the key associated with the data packet ID (recipient's or sender's depending on which data packet ID is used).
### Keep-alive command
To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the server responds with `pong` response. This command should be sent unsigned and without file chunk ID.
To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the router responds with `pong` response. This command should be sent unsigned and without data packet ID.
```abnf
ping = %s"PING"
@@ -405,21 +415,19 @@ ping = %s"PING"
This command is always sent unsigned.
data FileResponse = ... | FRPong | ...
```abnf
pong = %s"PONG"
```
### File sender commands
Sending any of the commands in this section (other than `register`, that is sent without file chunk ID) is only allowed with sender's ID.
Sending any of the commands in this section (other than `register`, that is sent without data packet ID) is only allowed with sender's ID. The `register` command must be signed (using `sndKey` included in `fileInfo` for verification) but must NOT include a data packet ID.
#### Register new file chunk
#### Register new data packet
This command is sent by the sender to the XFTP server to register a new file chunk.
This command is sent by the sender to the XFTP router to register a new data packet.
Servers SHOULD support basic auth with this command, to allow only server owners and trusted users to create file chunks on the servers.
Routers SHOULD support basic auth with this command, to allow only router owners and trusted users to create data packets on the routers.
The syntax is:
@@ -427,7 +435,7 @@ The syntax is:
register = %s"FNEW " fileInfo rcvPublicAuthKeys basicAuth
fileInfo = sndKey size digest
sndKey = length x509encoded
size = 1*DIGIT
size = 4*4 OCTET ; Word32 big-endian
digest = length *OCTET
rcvPublicAuthKeys = length 1*rcvPublicAuthKey
rcvPublicAuthKey = length x509encoded
@@ -438,7 +446,7 @@ x509encoded = <binary X509 key encoding>
length = 1*1 OCTET
```
If the file chunk is registered successfully, the server must send `sndIds` response with the sender's and recipients' file chunk IDs:
If the data packet is registered successfully, the router must send `sndIds` response with the sender's and recipients' data packet IDs:
```abnf
sndIds = %s"SIDS " senderId recipientIds
@@ -447,9 +455,9 @@ recipientIds = length 1*recipientId
recipientId = length *OCTET
```
#### Add file chunk recipients
#### Add data packet recipients
This command is sent by the sender to the XFTP server to add additional recipient keys to the file chunk record, in case number of keys requested by client didn't fit into `register` command. The syntax is:
This command is sent by the sender to the XFTP router to add additional recipient keys to the data packet record, in case number of keys requested by client didn't fit into `register` command. The syntax is:
```abnf
add = %s"FADD " rcvPublicAuthKeys
@@ -457,7 +465,7 @@ rcvPublicAuthKeys = length 1*rcvPublicAuthKey
rcvPublicAuthKey = length x509encoded
```
If additional keys were added successfully, the server must send `rcvIds` response with the added recipients' file chunk IDs:
If additional keys were added successfully, the router must send `rcvIds` response with the added recipients' data packet IDs:
```abnf
rcvIds = %s"RIDS " recipientIds
@@ -465,66 +473,100 @@ recipientIds = length 1*recipientId
recipientId = length *OCTET
```
#### Upload file chunk
#### Upload data packet
This command is sent by the sender to the XFTP server to upload file chunk body to server. The syntax is:
This command is sent by the sender to the XFTP router to upload data packet body to router. The syntax is:
```abnf
put = %s"FPUT"
```
Chunk body is streamed via HTTP2 request.
Packet body is streamed via HTTP2 request.
If file chunk body was successfully received, the server must send `ok` response.
If data packet body was successfully received, the router must send `ok` response.
```abnf
ok = %s"OK"
```
#### Delete file chunk
#### Delete data packet
This command is sent by the sender to the XFTP server to delete file chunk from the server. The syntax is:
This command is sent by the sender to the XFTP router to delete data packet from the router. The syntax is:
```abnf
delete = %s"FDEL"
```
Server should delete file chunk record, invalidating all recipient IDs, and delete file body from file storage. If file chunk was successfully deleted, the server must send `ok` response.
Router should delete data packet record, invalidating all recipient IDs, and delete file body from file storage. If data packet was successfully deleted, the router must send `ok` response.
### File recipient commands
Sending any of the commands in this section is only allowed with recipient's ID.
#### Download file chunk
#### Download data packet
This command is sent by the recipient to the XFTP server to download file chunk body from the server. The syntax is:
This command is sent by the recipient to the XFTP router to download data packet body from the router. The syntax is:
```abnf
get = %s"FGET " rDhKey
rDhKey = length x509encoded
```
If requested file is successfully located, the server must send `file` response. File chunk body is sent as HTTP2 response body.
If requested file is successfully located, the router must send `file` response. Data packet body is sent as HTTP2 response body.
```abnf
file = %s"FILE " sDhKey cbNonce
sDhKey = length x509encoded
cbNonce = <nonce used in NaCl crypto_box encryption scheme>
cbNonce = 24*24 OCTET ; NaCl crypto_box nonce
```
Chunk is additionally encrypted on the way from the server to the recipient using a key agreed via ephemeral DH keys `rDhKey` and `sDhKey`, so there is no ciphertext in common between sent and received traffic inside TLS connection, in order to complicate traffic correlation attacks, if TLS is compromised.
Packet is additionally encrypted on the way from the router to the recipient using a key agreed via ephemeral DH keys `rDhKey` and `sDhKey`, so there is no ciphertext in common between sent and received traffic inside TLS connection, in order to complicate traffic correlation attacks, if TLS is compromised.
#### Acknowledge file chunk download
#### Acknowledge data packet download
This command is sent by the recipient to the XFTP server to acknowledge file reception, deleting file ID from server for this recipient. The syntax is:
This command is sent by the recipient to the XFTP router to acknowledge file reception, deleting file ID from router for this recipient. The syntax is:
```abnf
ack = %s"FACK"
```
If file recipient ID is successfully deleted, the server must send `ok` response.
If file recipient ID is successfully deleted, the router must send `ok` response.
In current implementation of XFTP protocol in SimpleX Chat clients don't use FACK command. Files are automatically expired on servers after configured time interval.
In current implementation of XFTP protocol in SimpleX Chat clients don't use FACK command. Files are automatically expired on routers after configured time interval.
### Error responses
The router responds with `ERR` followed by the error type:
```abnf
error = %s"ERR " errorType
errorType = %s"BLOCK" / %s"SESSION" / %s"HANDSHAKE" /
%s"CMD" SP cmdError / %s"AUTH" / %s"BLOCKED" SP blockingInfo /
%s"SIZE" / %s"QUOTA" / %s"DIGEST" / %s"CRYPTO" /
%s"NO_FILE" / %s"HAS_FILE" / %s"FILE_IO" /
%s"TIMEOUT" / %s"INTERNAL"
cmdError = %s"UNKNOWN" / %s"SYNTAX" / %s"PROHIBITED" / %s"NO_AUTH" / %s"HAS_AUTH" / %s"NO_ENTITY"
blockingInfo = %s"reason=" blockingReason ["," %s"notice=" jsonNotice]
blockingReason = %s"spam" / %s"content"
jsonNotice = *OCTET ; JSON-encoded notice object
```
Error types:
- `BLOCK` - incorrect block format, encoding or signature size.
- `SESSION` - incorrect session ID (TLS Finished message / tls-unique binding).
- `HANDSHAKE` - incorrect handshake command.
- `CMD` - command syntax errors (UNKNOWN, SYNTAX, PROHIBITED, NO_AUTH, HAS_AUTH, NO_ENTITY).
- `AUTH` - command authorization error - bad signature or non-existing data packet.
- `BLOCKED` - data packet was blocked due to policy violation (added in v3). Contains blocking reason and optional notice.
- `SIZE` - incorrect file size.
- `QUOTA` - storage quota exceeded.
- `DIGEST` - incorrect file digest.
- `CRYPTO` - file encryption/decryption failed.
- `NO_FILE` - no expected file body in request/response or no file on the router.
- `HAS_FILE` - unexpected file body.
- `FILE_IO` - file IO error.
- `TIMEOUT` - file sending or receiving timeout.
- `INTERNAL` - internal router error.
## Threat model
@@ -533,7 +575,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
- A user protects their local database and key material.
- The user's application is authentic, and no local malware is running.
- The cryptographic primitives in use are not broken.
- A user's choice of servers is not directly tied to their identity or otherwise represents distinguishing information about the user.
- A user's choice of routers is not directly tied to their identity or otherwise represents distinguishing information about the user.
#### A passive adversary able to monitor the traffic of one user
@@ -541,7 +583,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
- identify that and when a user is sending files over XFTP protocol.
- determine which servers the user sends/receives files to/from.
- determine which routers the user sends/receives files to/from.
- observe how much traffic is being sent, and make guesses as to its purpose.
@@ -553,11 +595,11 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
*can:*
- learn which XFTP servers are used to send and receive files for which users.
- learn which XFTP routers are used to send and receive files for which users.
- learn when files are sent and received.
- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers.
- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers.
- observe how much traffic is being sent, and make guesses as to its purpose.
@@ -567,31 +609,31 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
- perform traffic correlation attacks.
#### XFTP server
#### XFTP router
*can:*
- learn when file senders and recipients are online.
- know how many file chunks and chunk sizes are sent via the server.
- know how many data packets and packet sizes are sent via the router.
- perform the correlation of the file chunks as belonging to one file via either a re-used transport connection, user's IP address, or connection timing regularities.
- perform the correlation of the data packets as belonging to one file via either a re-used transport connection, user's IP address, or connection timing regularities.
- learn file senders' and recipients' IP addresses, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used.
- delete file chunks, preventing file delivery, as long as redundant delivery is not used.
- delete data packets, preventing file delivery, as long as redundant delivery is not used.
- lie about the state of a file chunk to the recipient and/or to the sender (e.g. deleted when it is not).
- lie about the state of a data packet to the recipient and/or to the sender (e.g. deleted when it is not).
- refuse deleting the file when instructed by the sender.
*cannot:*
- undetectably corrupt file chunks.
- undetectably corrupt data packets.
- learn the contents, name or the exact size of sent files.
- learn approximate size of sent files, as long as more than one server is used to send file chunks.
- learn approximate size of sent files, as long as more than one router is used to send data packets.
- compromise the users' end-to-end encryption of files with an active attack.
@@ -603,7 +645,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
- receive all files sent and received by Alice that did not expire yet, as long as information about these files was not removed from the database.
- prevent Alice's contacts from receiving the files she sent by deleting all or some of the file chunks from XFTP servers.
- prevent Alice's contacts from receiving the files she sent by deleting all or some of the data packets from XFTP routers.
#### A user's contact
@@ -625,10 +667,10 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC
*can:*
- Denial of Service XFTP servers.
- Denial of Service XFTP routers.
*cannot:*
- send files to a user who they are not connected with.
- enumerate file chunks on an XFTP server.
- enumerate data packets on an XFTP router.
+13 -19
View File
@@ -10,7 +10,7 @@ Version 1, 2024-06-22
- [Session invitation](#session-invitation)
- [Establishing TLS connection](#establishing-tls-connection)
- [Session verification and protocol negotiation](#session-verification-and-protocol-negotiation)
- [Controller/host session operation](#сontrollerhost-session-operation)
- [Controller/host session operation](#controllerhost-session-operation)
- [Key agreement for announcement packet and for session](#key-agreement-for-announcement-packet-and-for-session)
- [Threat model](#threat-model)
@@ -104,12 +104,11 @@ Multicast session announcement is a binary encoded packet with this syntax:
```abnf
sessionAddressPacket = dhPubKey nonce encrypted(unpaddedSize sessionAddress packetPad)
dhPubKey = length x509encoded ; same as announced
nonce = length *OCTET
sessionAddress = largeLength sessionAddressUri ; as above
nonce = 24*24 OCTET ; NaCl 192-bit nonce, no length prefix
sessionAddress = sessionAddressUri ; length given by unpaddedSize
length = 1*1 OCTET ; for binary data up to 255 bytes
largeLength = 2*2 OCTET ; for binary data up to 65535 bytes
packetPad = <pad packet size to 1450 bytes> ; possibly, we may need to move KEM agreement one step later,
; with encapsulation key in HELLO block and KEM ciphertext in reply to HELLO.
packetPad = <pad invitation content to 900 bytes before encryption>
```
### Establishing TLS connection
@@ -143,7 +142,7 @@ hostHello = %s"HELLO " dhPubKey nonce encrypted(unpaddedSize hostHelloJSON hello
unpaddedSize = largeLength
dhPubKey = length x509encoded
pad = <pad block size to 16384 bytes>
helloPad = <pad hello size to 12888 bytes>
helloPad = <pad hello size to 12288 bytes>
largeLength = 2*2 OCTET
```
@@ -157,10 +156,7 @@ The controller decrypts (including the first session) and validates the received
{
"definitions": {
"version": {
"type": "string",
"metadata": {
"format": "[0-9]+"
}
"type": "uint16"
},
"base64url": {
"type": "string",
@@ -172,9 +168,7 @@ The controller decrypts (including the first session) and validates the received
"properties": {
"v": {"ref": "version"},
"ca": {"ref": "base64url"},
"kem": {"ref": "base64url"}
},
"optionalProperties": {
"kem": {"ref": "base64url"},
"app": {"properties": {}, "additionalProperties": true}
},
"additionalProperties": true
@@ -190,7 +184,7 @@ ctrlHello = %s"HELLO " kemCiphertext encrypted(unpaddedSize ctrlHelloJSON helloP
unpaddedSize = largeLength
kemCiphertext = largeLength *OCTET
pad = <pad block size to 16384 bytes>
helloPad = <pad hello size to 12888 bytes>
helloPad = <pad hello size to 12288 bytes>
largeLength = 2*2 OCTET
ctrlError = %s"ERROR " nonce encrypted(unpaddedSize ctrlErrorMessage helloPad) pad
@@ -206,7 +200,7 @@ JTD schema for the encrypted part of controller HELLO block `ctrlHelloJSON`:
}
```
Controller `hello` block and all subsequent protocol messages are encrypted with the chain keys derived from the hybrid key (see key exchange below) - that is why conntroller hello block does not include nonce. That provides forward secrecy within the XRCP session. Receiving this `hello` block allows host to compute the same hybrid keys and to derive the same chain keys.
Controller `hello` block and all subsequent protocol messages are encrypted with the chain keys derived from the hybrid key (see key exchange below) - that is why controller hello block does not include nonce. That provides forward secrecy within the XRCP session. Receiving this `hello` block allows host to compute the same hybrid keys and to derive the same chain keys.
Once the controller replies HELLO to the valid host HELLO block, it should stop accepting new TCP connections.
@@ -261,7 +255,7 @@ kemCiphertext(1) = enc(kemSecret(1), kemEncKey(1))
kemSecret(1) = dec(kemCiphertext(1), kemDecKey(1))
// multicast announcement for session n
announcementSecret(n) = sha256(dhSecret(n'))
announcementSecret(n) = dhSecret(n')
dhSecret(n') = dh(hostHelloDhKey(n - 1), controllerDhKey(n))
// session n
@@ -277,11 +271,11 @@ If controller fails to store the new host DH key after receiving HELLO block, th
To decrypt a multicast announcement, the host should try to decrypt it using the keys of all known (paired) remote controllers.
Once kemSecret is agreed for the session, it is used to derive two chain keys, to receive and to send messages:
Once sessionSecret is agreed for the session, it is used to derive two chain keys, to receive and to send messages:
```
host: sndKey, rcvKey = HKDF(kemSecret, "SimpleXSbChainInit", 64)
controller: rcvKey, sndKey = HKDF(kemSecret, "SimpleXSbChainInit", 64)
controller: sndKey, rcvKey = HKDF(sessionSecret, "SimpleXSbChainInit", 64)
host: rcvKey, sndKey = HKDF(sessionSecret, "SimpleXSbChainInit", 64)
```
where HKDF is based on SHA512, with empty salt.
+2 -2
View File
@@ -3,9 +3,9 @@
## Problem
When sending an SMP confirmation a network timeout can lead to the following race condition:
- server receives the confirmation while the joining party fails to receive the server's response;
- router receives the confirmation while the joining party fails to receive the router's response;
- joining party deletes the connection together with credentials sent in the confirmation for securing the queue;
- initiating party will receive the confirmation from the server and secure the queue;
- initiating party will receive the confirmation from the router and secure the queue;
- on subsequent attempt to join via the same invitation link initiating party will generate new credentials and fail authorization.
This renders the joining party permanently unable to join via that invitation link and complete the connection.
+2 -2
View File
@@ -3,12 +3,12 @@
## Problem
iOS notifications may fail to deliver for several reasons, but there are two important reasons that we could address:
- when notification server is not subscribed to SMP server(s), the notifications can be dropped - it can happen because either notification server restarts or becuase SMP server restarted and some messages are received before notification server resubscribed. We lose approximately 3% of notifications because of this reason.
- when notification router is not subscribed to SMP router(s), the notifications can be dropped - it can happen because either notification router restarts or becuase SMP router restarted and some messages are received before notification router resubscribed. We lose approximately 3% of notifications because of this reason.
- when user device is offline or has low power condition, Apple does not deliver notification, but puts them to storage. If while the notification is in storage a new one arrives it would overwrite the previous notification. If it was the message to the same message queue, the client will download messages anyway, up to a limit, but if the message was to another queue, it will not be delivered until the app is opened. Apple delivers about 88% of notifications that should be delivered (not accounting for uninstalled apps), the rest is replaced with the newer notifications.
## Solution
The first problem can be solved by preserving notifications for a limited time (say 1 hour) in case there is no subscription to notification from notification server. At the very least, they can be preserved in SMP server memory but can also be stored to a file on restart, similar to messages, and be delivered when notification server resubscribes. It is sufficient to store one notification per messaging queue.
The first problem can be solved by preserving notifications for a limited time (say 1 hour) in case there is no subscription to notification from notification router. At the very least, they can be preserved in SMP router memory but can also be stored to a file on restart, similar to messages, and be delivered when notification router resubscribes. It is sufficient to store one notification per messaging queue.
The second problem is both more damaging and more complex to solve. The solution could be to always deliver several last notifications to different queues in one packet (Apple allows up to ~4-5kb notification size, and we are sending packets of fixed size 512 bytes, so we could fit up to 8-10 of them in each notification).
+12 -12
View File
@@ -8,7 +8,7 @@ See [Short invitation links](./2024-06-21-short-links.md).
2) clients only delete queue records based on some user action, pending connections do not expire.
While part 2 should be improved in the client, indefinite storage of queue records becomes a much bigger issue if each of them would result in a permanent storage of 4-16kb blob in server memory, without server-side expiration for short invitation links.
While part 2 should be improved in the client, indefinite storage of queue records becomes a much bigger issue if each of them would result in a permanent storage of 4-16kb blob in router memory, without router-side expiration for short invitation links.
## Possible solutions
@@ -16,15 +16,15 @@ While part 2 should be improved in the client, indefinite storage of queue recor
The problem with this approach is that contact addresses are also unsecured queues, and they should not be expired.
We could set really large expiration time, and require that clients "update" the unsecured queues they need at least every 1-2 years, but it would not solve the problem of storing a large number of blobs in the server memory for unused/abandoned 1-time invitations.
We could set really large expiration time, and require that clients "update" the unsecured queues they need at least every 1-2 years, but it would not solve the problem of storing a large number of blobs in the router memory for unused/abandoned 1-time invitations.
2) Do not store blobs in memory / append-only log, and instead use something like RocksDB. While it may be a correct long term solution, it may be not expedient enough at the current POC stage for this feature. Also, the lack of expiration is wrong in any case and would indefinitely grow server storage.
2) Do not store blobs in memory / append-only log, and instead use something like RocksDB. While it may be a correct long term solution, it may be not expedient enough at the current POC stage for this feature. Also, the lack of expiration is wrong in any case and would indefinitely grow router storage.
3) Add flag allowing the server to differentiate permanent queues used as contact addresses, also using different blob sizes for them. In this case, messaging queues will be expired if not secured after 3 weeks, and contact address queues would be expired if not "updated" by the owner within 2 years.
3) Add flag allowing the router to differentiate permanent queues used as contact addresses, also using different blob sizes for them. In this case, messaging queues will be expired if not secured after 3 weeks, and contact address queues would be expired if not "updated" by the owner within 2 years.
Probably all three solutions need to be used, to avoid creating a non-expiring blob storage in memory, as in case too many of such blobs are created it would not be possible to differentiate between real users and resource exhaustion attacks, and unlike with messages, they won't be expiring too.
Servers already can differentiate messaging queues and contact address queues, if they want to:
Routers already can differentiate messaging queues and contact address queues, if they want to:
- with the old 4-message handshake, the confirmation message on a normal queue was different, and also KEY command was eventually used.
- with the fast 2-message handshake, while the confirmation message has the same syntax, and the differences are inside encrypted envelope, the client still uses SKEY command.
- in both cases, the usual messaging queues are secured, and contact addresses are not, so this difference is visible in the storage as well (although it is not easy to differentiate between abandoned 1-time invitations and contact addresses).
@@ -33,7 +33,7 @@ Differentiating these queues can also allow different message retention times -
## Proposed solution
1. Add queue updated_at date into queue records. While it adds some metadata, it seems necessary to manage retention and quality of service. It will not include exact time, only date, and the time of creation will be replaced by the time of any update - queue secured, a message is sent, or queue owner subscribes to the queue. To avoid the need to update store log on every message this information can be appended to store log on server termination. Or given that only one update per day is needed it may be ok to make these updates as they happen (temporarily making the sequence and time of these events available in storage).
1. Add queue updated_at date into queue records. While it adds some metadata, it seems necessary to manage retention and quality of service. It will not include exact time, only date, and the time of creation will be replaced by the time of any update - queue secured, a message is sent, or queue owner subscribes to the queue. To avoid the need to update store log on every message this information can be appended to store log on router termination. Or given that only one update per day is needed it may be ok to make these updates as they happen (temporarily making the sequence and time of these events available in storage).
2. Add flag to indicate the queue usage - messaging queue or queue for contact address connection requests. This would result in different queue size and different retention policy for queue and its messages. We already have "sender can secure flag" which is, effectively, this flag - contact address queues are never secured. So this does not increase stored metadata in any way.
@@ -41,11 +41,11 @@ Differentiating these queues can also allow different message retention times -
This is a design considerations and a concept, not a design yet.
Instead of implementing a generic blob storage that can be used as an attack vector, and adds additional failure point (another server storing blob that is necessary to connect to the queue on the current server), but instead adds an extended queue information blobs, most of which could be dropped without the loss of connectivity, so that the attack can be mitigated by deleting these blobs without users losing the ability to connect, as long as the queue and minimal extended information is retained.
Instead of implementing a generic blob storage that can be used as an attack vector, and adds additional failure point (another router storing blob that is necessary to connect to the queue on the current router), but instead adds an extended queue information blobs, most of which could be dropped without the loss of connectivity, so that the attack can be mitigated by deleting these blobs without users losing the ability to connect, as long as the queue and minimal extended information is retained.
So, to make the connection there need to be these elements:
- queue server and queue ID - mandatory part, that can be included in short link
- queue router and queue ID - mandatory part, that can be included in short link
- SMP key - mandatory part for all queues. We are considering initializing ratchets earlier for contact addresses, and include ratchet keys and pre-keys into queue data as well, but it is out of scope here.
- Ratchet keys - mandatory part for 1-time invitation that won't fit in short link.
- PQ key - optional part that can be stored with addresses if ratchet keys are added and with 1-time invitations.
@@ -56,8 +56,8 @@ So rather that storing one blob with a large address inside it, not associated w
Also, we need the address shared with the sender (party accepting the connection) to be short. We could use a similar approach that was proposed for data blobs, using a single random seed per queues to derive multiple keys and IDs from it. For example:
1. The queue owner:
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now sent in NEW command.
- generates queue recipient ID (this ID can still be server-generated).
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the router, same as now sent in NEW command.
- generates queue recipient ID (this ID can still be router-generated).
- generates X25519 key pair `(k, pk)` to use with the accepting party.
- derives from `k`:
- sender ID.
@@ -73,9 +73,9 @@ The algorithm used to derive key and ID from `k` needs to be cryptographically s
So, coupling blob storage with messaging queues has these pros/cons:
Cons:
- no additional layer of privacy - the server used for connection is visible in the link, even after the blobs are removed from the server.
- no additional layer of privacy - the router used for connection is visible in the link, even after the blobs are removed from the router.
Pros:
- no additional point of failure in the connection process - the same server will be used to retrieve necessary blobs as for connection.
- no additional point of failure in the connection process - the same router will be used to retrieve necessary blobs as for connection.
- queue blobs of messaging blobs will be automatically removed once the queue is secured or expired, without additional request from the recipient - reducing the storage and the time these blobs are available.
- queue blobs for contact addresses will be structured and some of the large blobs can be removed in case of resource exhaustion attack (and recreated by the client if needed), with the only downside that PQ handshake will be postponed (which is the case now) and profile will not be available at a point of connection.
+8 -8
View File
@@ -2,25 +2,25 @@
## Problem
Our current handshake protocol is open to this attack: whoever observes the link exchange, knows on which server connection is being made, and if the traffic on this server is observed, then it can confirm communication between parties. Further, even with the [last proposal](./2024-09-09-smp-blobs.md#possible-privacy-improvement), having real-time access to the server data allows to establish the exact messaging queue that is used to send messages.
Our current handshake protocol is open to this attack: whoever observes the link exchange, knows on which router connection is being made, and if the traffic on this router is observed, then it can confirm communication between parties. Further, even with the [last proposal](./2024-09-09-smp-blobs.md#possible-privacy-improvement), having real-time access to the router data allows to establish the exact messaging queue that is used to send messages.
## Solution
We could make the initial link exchange more private by making it harder for any observer to discover which server will be used for messaging by hiding this information from the server that hosts the initial link.
We could make the initial link exchange more private by making it harder for any observer to discover which router will be used for messaging by hiding this information from the router that hosts the initial link.
Preliminary, the protocol could be the following:
1. Connection initiator stores 224-256 bytes of encrypted connection link on a rendezvous server (link contains server host and linkId on another messaging server, not a rendezvous one).
1. Connection initiator stores 224-256 bytes of encrypted connection link on a rendezvous router (link contains router host and linkId on another messaging router, not a rendezvous one).
2. Rendezvous server adds these links to buckets, up to 64 links per bucket. Bucket ID is the timestamp when the bucket was created + a sequential bucket number, in case more than one bucket is created per second.
2. Rendezvous router adds these links to buckets, up to 64 links per bucket. Bucket ID is the timestamp when the bucket was created + a sequential bucket number, in case more than one bucket is created per second.
3. The server responds to the link creator with a bucket ID where this link was added. That bucket ID is its timestamp + a number prevents server "fingerprinting" clients and using say one bucket for each client. If timestamp is different or a bucket number within this timestamp is too large, the client can refuse to use it, depending on the client settings.
3. The router responds to the link creator with a bucket ID where this link was added. That bucket ID is its timestamp + a number prevents router "fingerprinting" clients and using say one bucket for each client. If timestamp is different or a bucket number within this timestamp is too large, the client can refuse to use it, depending on the client settings.
4. The initiating party will pass to the accepting party the rendezvous server host, the hash of this bucket ID (bucket link) and the passphrase to derive the key from. The initiating party has an option to pass a link and passphrase via two channels - in which case the link will only contain the bucket ID.
4. The initiating party will pass to the accepting party the rendezvous router host, the hash of this bucket ID (bucket link) and the passphrase to derive the key from. The initiating party has an option to pass a link and passphrase via two channels - in which case the link will only contain the bucket ID.
5. The accepting party would then request the bucket via its ID hash (the server would store hashes to be able to look up - hash is used to prevent showing time in the link) and attempt to decrypt all contained links using the provided key.
5. The accepting party would then request the bucket via its ID hash (the router would store hashes to be able to look up - hash is used to prevent showing time in the link) and attempt to decrypt all contained links using the provided key.
The accepting party then will continue the connection via the decrypted link.
This obviously does not protect accepting party from the initiating party, if it can choose rendezvous server it controls. It also does not protect from the malicious rendezvous server that would collaborate with link observers. I think reunion doesnt protect from it too.
This obviously does not protect accepting party from the initiating party, if it can choose rendezvous router it controls. It also does not protect from the malicious rendezvous router that would collaborate with link observers. I think reunion doesnt protect from it too.
But it does protect connection from whoever observes the link, particularly if this link only contains the bucket and the key is passed separately, via some other channel.
+9 -9
View File
@@ -2,7 +2,7 @@
## Problem
For iOS notifications to be delivered the client has to create credentials for notification subscription on SMP server using NKEY command and after that create a subscription on notification server using SNEW command. These two commands are sent in sequence, after the connections are created, and for it to happen the client needs to be online and in foreground.
For iOS notifications to be delivered the client has to create credentials for notification subscription on SMP router using NKEY command and after that create a subscription on notification router using SNEW command. These two commands are sent in sequence, after the connections are created, and for it to happen the client needs to be online and in foreground.
iOS users tend to close the app when it is not used, and iOS has very limited permissions for background activities, so these notification subscriptions are created with a substantial delay, and notifications do not work.
@@ -12,19 +12,19 @@ This problem is distinct from and probably more common than other problems affec
1. When the new connection is created, the client already knows if it needs to create notification subscription or not, based on the conversation setting (e.g., if the group is muted, the client will not create notification subscription as well.). We should extend NEW command to avoid the need to send additional NKEY command with an option to create notification subscription at the point where connection is created. NDEL would still be used to disable this notification, and NKEY will be used to re-enable it.
2. In the same way we stopped using SDEL command (NDEL sends notification DELD to subscribed notification server) to delete notificaiton subscriptions from notification server, we should delegate creating notification subscription on notification server to SMP servers. Clients could use keys agreed with ntf server for e2e encryption and for command authorization to encrypt and sign instruction to create notification subscription that will be forwarded to notification server using protocol similar to SMP proxies. This will avoid the need for clients to separately contact notification servers that won't happen until they are online.
2. In the same way we stopped using SDEL command (NDEL sends notification DELD to subscribed notification router) to delete notificaiton subscriptions from notification router, we should delegate creating notification subscription on notification router to SMP routers. Clients could use keys agreed with ntf router for e2e encryption and for command authorization to encrypt and sign instruction to create notification subscription that will be forwarded to notification router using protocol similar to SMP proxies. This will avoid the need for clients to separately contact notification routers that won't happen until they are online.
3. Instead of making Ntf server trust DELD notifications, we could send deletion instructions signed by the client, which will only fail to send in case notification server is down (and they won't be sent later after server restart).
3. Instead of making Ntf router trust DELD notifications, we could send deletion instructions signed by the client, which will only fail to send in case notification router is down (and they won't be sent later after router restart).
Cons:
- If SMP servers were to retain in the storage the information about which notification server is used for which queue, it would reduce metadata privacy. While currently it is not an issue, as all notification servers are known and operated by us, once there are other client apps, this can be used for app users fingerprinting, which would act as a deterrence from using new apps but only if app users use servers of operators who are different from the app provider. To mitigate it, we could only store it in server memory and include notification instruction in subscription commands (SUB) and include notification subscription status in SUB responses. We don't need to mitigate the problem of server being able to store this information, as messaging servers can observe which notification servers connect to them anyway.
- If SMP server is restarted before the subscription request is forwared to the notification server, then it will have to be forwarded again, once the client subscribes. The problem here is that if the client is offline, it will neither subscribe to the queue to send notification subscription request, nor receive notifications from this queue. Storing notification server and subscription request would mitigate that, as in this case we could send all pending requests on server start, without depending on client subscriptions.
- "Small" agent will need to support connections to ntf servers and manage workers that retry sending pending subscription requests.
- Until the client learns the public keys of notification server, it will not be able to decrypt notifications. It potentially can be mitigated by using the public key of the server returned when token is created, in this way different client keys (per-queue) will be combined with the same ntf server key (per-token).
- If SMP routers were to retain in the storage the information about which notification router is used for which queue, it would reduce metadata privacy. While currently it is not an issue, as all notification routers are known and operated by us, once there are other client apps, this can be used for app users fingerprinting, which would act as a deterrence from using new apps but only if app users use routers of operators who are different from the app provider. To mitigate it, we could only store it in router memory and include notification instruction in subscription commands (SUB) and include notification subscription status in SUB responses. We don't need to mitigate the problem of router being able to store this information, as messaging routers can observe which notification routers connect to them anyway.
- If SMP router is restarted before the subscription request is forwared to the notification router, then it will have to be forwarded again, once the client subscribes. The problem here is that if the client is offline, it will neither subscribe to the queue to send notification subscription request, nor receive notifications from this queue. Storing notification router and subscription request would mitigate that, as in this case we could send all pending requests on router start, without depending on client subscriptions.
- "Small" agent will need to support connections to ntf routers and manage workers that retry sending pending subscription requests.
- Until the client learns the public keys of notification router, it will not be able to decrypt notifications. It potentially can be mitigated by using the public key of the router returned when token is created, in this way different client keys (per-queue) will be combined with the same ntf router key (per-token).
## Implementation details
1. NEW and NKEY commands will need to be extended to include notification subscription request. As the notifier ID needs to be sent to notification server, this notifier ID will have to be client-generated and supplied as part of NEW command.
1. NEW and NKEY commands will need to be extended to include notification subscription request. As the notifier ID needs to be sent to notification router, this notifier ID will have to be client-generated and supplied as part of NEW command.
now:
@@ -46,4 +46,4 @@ NKEY :: NtfPublicAuthKey -> RcvNtfPublicDhKey -> Maybe NtfServerRequest -> Comma
-- NotifierID is passed in entity ID field of the transmission
```
2. Notification server will need to support an additional command to receive "proxied" subscription commands, `SFWD`, that would include `NtfServerRequest`. This command can include both `SNEW` and `SDEL` commands.
2. Notification router will need to support an additional command to receive "proxied" subscription commands, `SFWD`, that would include `NtfServerRequest`. This command can include both `SNEW` and `SDEL` commands.
+1 -1
View File
@@ -5,7 +5,7 @@ This document evolves the design proposed [here](./2024-09-09-smp-blobs.md).
## Problems
In addition to problems in the first doc, we have these issues with in-memory queue record storage:
- many queues are idle or rarely used, but they are loaded to memory, and currently just loading all queues uses 20gb RAM on each server, and takes 10 min to process, increasing downtimes during restarts.
- many queues are idle or rarely used, but they are loaded to memory, and currently just loading all queues uses 20gb RAM on each router, and takes 10 min to process, increasing downtimes during restarts.
- adding blobs to memory would make this problem much worse.
## Proposed solution
+21 -21
View File
@@ -6,65 +6,65 @@ iOS notifications have these problems:
- iOS notification service crashes exceeding memory limit. This is being addressed by changes in GHC RTS.
- there is a large number of connections, because each member in a group requires individual connection. This will improve with chat relays when each group would require 2-3 connections.
- some notification may be not shown if notification with reply/mention is skipped, and instead some other message is delivered, which may be muted. This would not improve without some changes, as notifications may be skipped anyway.
- client devices delay communication with ntf server because it is done in background, and by that time the app may be suspended.
- notification server represents a bottleneck, as it has to be owned by the app vendor, and the current design when ntf server subscribes to notifications scales very badly.
- client devices delay communication with ntf router because it is done in background, and by that time the app may be suspended.
- notification router represents a bottleneck, as it has to be owned by the app vendor, and the current design when ntf router subscribes to notifications scales very badly.
This RFC is based on the previous [RFC related to notifications](./2024-09-25-ios-notifications-2.md).
## Solution
As notification server has to know client token and currently it associates subscriptions with this token anyway, we are not gaining any privacy and security by using per-subscription keys - both authorization and encryption keys of notification subscription can be dropped.
As notification router has to know client token and currently it associates subscriptions with this token anyway, we are not gaining any privacy and security by using per-subscription keys - both authorization and encryption keys of notification subscription can be dropped.
We still need to store the list of queue IDs associated with the token on the notification server, but we do not need any per-queue keys on the notification server, and we don't need subscriptions - it's effectively a simple set of IDs, with no other information.
We still need to store the list of queue IDs associated with the token on the notification router, but we do not need any per-queue keys on the notification router, and we don't need subscriptions - it's effectively a simple set of IDs, with no other information.
In this case, when queue is created the client would supply notifier ID - it has to be derived from correlation ID, to prevent existense check (see previous RFC). As we also supply sender ID, instead of deriving it as sha3-192 of correlation ID, they both can be derived as sha3-384 and split to two IDs - 24 bytes each.
The notification server will maintain a rotating list of server keys with the latest key communicated to the client every time the token is registered and checked. The keys would expire after, say, 1 week or 1 month, and removed from notification server on expiration.
The notification router will maintain a rotating list of router keys with the latest key communicated to the client every time the token is registered and checked. The keys would expire after, say, 1 week or 1 month, and removed from notification router on expiration.
The packet containing association between notifier queue ID and token will be crypto_box encrypted using key agreement between identified notification server master key and an ephemeral per packet (effectively, per-queue) client-key.
The packet containing association between notifier queue ID and token will be crypto_box encrypted using key agreement between identified notification router master key and an ephemeral per packet (effectively, per-queue) client-key.
Deleting the queue may also include encrypted packet that would verify that the client deleted the queue.
Instead of notification server subscribing to the notifications creating a lot of traffic for the queues without messages, the SMP server would push notifications via NTF server connection (whether via NTF or via SMP protocol). This could be used as a mechanism to migrate existing queues when with the next subscription the notification server would communicate it's address to SMP server and this association would be stored together with the queue.
Instead of notification router subscribing to the notifications creating a lot of traffic for the queues without messages, the SMP router would push notifications via NTF router connection (whether via NTF or via SMP protocol). This could be used as a mechanism to migrate existing queues when with the next subscription the notification router would communicate it's address to SMP router and this association would be stored together with the queue.
## Protocol design
Additional/changed SMP commands:
```haskell
-- register notification server
-- should be signed with server key
-- register notification router
-- should be signed with router key
NSRV :: NtfServerCreds -> Command NtfServer
-- response
NSID :: NtfServerId -> BrokerMsg
-- to communicate which server is responsible for the queue
-- to communicate which router is responsible for the queue
-- should be signed with queue key
NSUB :: Maybe NtfServerId -> Command Notifier
-- subscribe to notificaions from all queues associated with the server
-- should be signed with server key
-- subscribe to notificaions from all queues associated with the router
-- should be signed with router key
-- entity ID - NtfServerId
NSSUB :: Command NtfServer
data NtfServerCreds = NtfServerCreds
{ server :: NtfServer,
-- NTF server certificate chain that should match fingerpring in address
-- NTF router certificate chain that should match fingerpring in address
cert :: X.CertificateChain,
-- server autorizatio key to sign server subscription requests
-- router autorizatio key to sign router subscription requests
authKey :: X.SignedExact X.PubKey
}
-- entity ID is recipient ID
NSKEY :: NtfSubscription -> Command Recipient
NSKEY :: NtfSubscription -> Command Recipient
data NtfSubscription = NtfSubscription
-- key to encrypt notifications e2e with the client
{ ntfPubDbKey :: RcvNtfPublicDhKey,
ntfServer :: NtfServer,
-- should be linked to correlation ID to prevent existense check
-- the ID sent to notification server could be its hash?
-- the ID sent to notification router could be its hash?
ntfId :: NotifierId,
encNtfTokenAssoc :: EncDataBytes
}
@@ -77,12 +77,12 @@ data NtfTokenAssoc = NtfTokenAssoc
}
```
SMP server will need to maintain the list of Ntf servers and their credentials, and when NSSUB arrives to make only one subscription. When message arrives it would deliver notification to the correct connection via queue / ntf server association.
SMP router will need to maintain the list of Ntf routers and their credentials, and when NSSUB arrives to make only one subscription. When message arrives it would deliver notification to the correct connection via queue / ntf router association.
Ntf server needs to maintain three indices to the same data:
Ntf router needs to maintain three indices to the same data:
- `(smpServer, queueId) -> tokenId` - to deliver notification to the correct token
- `tokenId -> [smpServer -> [queueId]]` - to remove all queues when token is removed, and to store/update these associations effficiently - store log may have one compact line per token (after compacting), or per token/server combination.
- `[smpServer]` - array of SMP servers to subscribe to.
- `tokenId -> [smpServer -> [queueId]]` - to remove all queues when token is removed, and to store/update these associations effficiently - store log may have one compact line per token (after compacting), or per token/router combination.
- `[smpServer]` - array of SMP routers to subscribe to.
## Mention notifications
@@ -90,4 +90,4 @@ Currently we are marking messages with T (true) for messages that require notifi
The proposal is to:
- add additional values to this metadata, e.g. 2 (priority) and 3 (high priority) (and T/F could be sent as 0/1 respectively) - that is, to deliver notifications even if notifications are generally disabled (they can still be further filtered by the client).
- instead of deleting notification credentials when notifications are disabled - which is costly - communicate to SMP server the change of notificaion priority level, e.g. the client could set minimal notification priority to deliver notifications, where 0 would mean disabling it completely, 1 enable for all, 2 for priority 2+, 3 for priority 3. The downside here is that it could be used for timing correlation of queues in the group, but it already can be used on bulk deletions of ntf credentials for these queues and when sending messages.
- instead of deleting notification credentials when notifications are disabled - which is costly - communicate to SMP router the change of notificaion priority level, e.g. the client could set minimal notification priority to deliver notifications, where 0 would mean disabling it completely, 1 enable for all, 2 for priority 2+, 3 for priority 3. The downside here is that it could be used for timing correlation of queues in the group, but it already can be used on bulk deletions of ntf credentials for these queues and when sending messages.
+9 -9
View File
@@ -35,18 +35,18 @@ This could possibly be evolved into the requirement to have a direct connection
3. Allow "joint management" of SMP queues.
SMP servers can support multiple recipients for contact queues:\
SMP routers can support multiple recipients for contact queues:\
- subscription would be possible to the "subscriber recipient".
- all other changes (update data, change subscriber recipient, add or remove recipients) would require multiple recipient signatures on SMP command in line with n-of-m multisig rules, that the command sender would have to collect out-of-band (from SMP protocol point of view).
Pros: allows joint ownership, and protects from losing access to master owner device.
Cons:
- complicates queue abstraction with approach that is not needed for most queues.
- still retains the server as a single point of failure.
- still retains the router as a single point of failure.
4. Introduce "group" as a new type of entity managed by SMP servers.
4. Introduce "group" as a new type of entity managed by SMP routers.
SMP servers would provide a separate set of commands for managing group records that would include in an encrypted container:
SMP routers would provide a separate set of commands for managing group records that would include in an encrypted container:
- the group profile
- the list of chat relay links
- the list of owner member IDs with their public keys
@@ -54,7 +54,7 @@ SMP servers would provide a separate set of commands for managing group records
- alternative group entity locations
- possibly, a globally unique group identity (as the hash of the initial/seed group data).
While the server domain would be used as the hostname in group link, it may contain alternative hosts (not just hostnames of the same server), both in the link and in the group record data.
While the router domain would be used as the hostname in group link, it may contain alternative hosts (not just hostnames of the same router), both in the link and in the group record data.
Pros: separates additional complexity to where it is needed, allowing reliability and redundancy for group ownership.
Cons: complexity, coupling between SMP and chat protocol.
@@ -86,7 +86,7 @@ Cons:
- if no messages are accepted, this is not even a queue.
- no way to directly contact owners (maybe it is not a downside, as for relays there would be a communication channel anyway as part of the group).
Option 2 looks more simple and attractive, implementing server broadcast for SMP seems unnecessary, as while it could have been used for simple groups, it does not solve such problems as spam and pre-moderation anyway - it requires a higher level protocol.
Option 2 looks more simple and attractive, implementing router broadcast for SMP seems unnecessary, as while it could have been used for simple groups, it does not solve such problems as spam and pre-moderation anyway - it requires a higher level protocol.
The command to update owner keys would be `RKEY` with the list of keys, and we can make `NEW` accept multiple keys too, although the use case here is less clear.
@@ -96,7 +96,7 @@ Option 1: Use the same keys in SMP as when signing queue data.
Option 2: Use different keys.
The value here could be that the server could validate these signatures too, and also maintain the chain of key changes. While tempting, it is probably unnecessary, and this chain of ownership is better to be maintained on chat relay level, as there are no size constraints on the size of this chain. Also, it is better for metadata privacy to not couple transport and chat protocol keys.
The value here could be that the router could validate these signatures too, and also maintain the chain of key changes. While tempting, it is probably unnecessary, and this chain of ownership is better to be maintained on chat relay level, as there are no size constraints on the size of this chain. Also, it is better for metadata privacy to not couple transport and chat protocol keys.
We still need to bind the mutable data updates to the "genesis" signature key (the one included in the immutable data).
@@ -147,12 +147,12 @@ The size of the OwnerInfo record encoding is:
~189 bytes, so we should practically limit the number of owners to say 8 - 1 original + 7 addiitonal. Original creator could use a different key as a "genesis" key, to conceal creator identity from other members, and it needs to include the record with memberId anyway.
The structure is simplified, and it does not allow arbitrary ownership changes. Its purpose is not to comprehensively manage ownership changes - while it is possible with a generic blockchain, it seems not appropriate at this stage, - but rather to ensure access continuity and that the server cannot modify the data (although nothing prevents the server from removing the data completely or from serving the previous version of the data).
The structure is simplified, and it does not allow arbitrary ownership changes. Its purpose is not to comprehensively manage ownership changes - while it is possible with a generic blockchain, it seems not appropriate at this stage, - but rather to ensure access continuity and that the router cannot modify the data (although nothing prevents the router from removing the data completely or from serving the previous version of the data).
For example it would only allow any given owner to remove subsequenty added owners, preserving the group link and identity, but it won't allow removing owners that signed this owner authorization. So owners are not equal, with the creator having the highest rank and being able to remove all additional owners, and owners authorise by creator can remove all other owners but themselves and creator, and so on - they have to maintain the chain that authorized themselves, at least. We could explicitely include owner rank into OwnerInfo, or we could require that they are sorted by rank, or the rank can be simply derived from signatures.
When additional owners want to be added to the group, they would have to provide any of the current owners:
- the key for SMP commands authorization - this will be passed to SMP server together with other keys. There could be either RKEY to pass all keys (some risk to miss some, or of race conditions), or RADD/RGET/RDEL to add and remove recipient keys, which has no risk of race conditions.
- the key for SMP commands authorization - this will be passed to SMP router together with other keys. There could be either RKEY to pass all keys (some risk to miss some, or of race conditions), or RADD/RGET/RDEL to add and remove recipient keys, which has no risk of race conditions.
- the signature of the immutable data by their member key included in their profile.
- the current owner would then include their member key into the queue data, and update it with LSET command. In any case there should be some simple consensus protocol between owners for owner changes, and it has to be maintained as a blockchain by owners and by chat relays, as otherwise it may lead to race conditions with LSET command.
+14 -14
View File
@@ -12,13 +12,13 @@ In addition to that, the specific implementation of this approach in Signal comp
While this limitation can be addressed with notifications when a new device is added and per-device keys, we still find the remaining attack vectors on user security and privacy to be unacceptable, and opening unsuspecting users to various criminal actions - and it is wrong to say that would only affect security conscious users, and most people would not be affected by these risks. Allowing potential criminals in groups to know which device you are currently using is a real risk for all users.
Another approach was offered by Threema that is ["mediator" server](https://threema.com/en/blog/md-architectural-overview) where the state of encryption ratchets is stored server-side. While it protects the user from their communication peers, it increases required level of trust to the servers, and in case of SimpleX network it would expose the knowledge of who communicates to whom. So while the idea of server-side storage of encryption state is promising, it has to be per-connection, to retain "no-accounts" property of SimpleX messaging network.
Another approach was offered by Threema that is ["mediator" router](https://threema.com/en/blog/md-architectural-overview) where the state of encryption ratchets is stored router-side. While it protects the user from their communication peers, it increases required level of trust to the routers, and in case of SimpleX network it would expose the knowledge of who communicates to whom. So while the idea of router-side storage of encryption state is promising, it has to be per-connection, to retain "no-accounts" property of SimpleX messaging network.
Also see [FAQ](https://simplex.chat/faq/#why-cant-i-use-the-same-profile-on-different-devices) and [this issue](https://github.com/simplex-chat/simplex-chat/issues/444#issuecomment-3066968358).
## Proposed solution
One of the ideas presented in FAQ - to store the state of Double Ratchet algorithm in the encrypted container on the server seems promising. The RFC develops this idea.
One of the ideas presented in FAQ - to store the state of Double Ratchet algorithm in the encrypted container on the router seems promising. The RFC develops this idea.
### Considerations for the design
@@ -26,21 +26,21 @@ One of the ideas presented in FAQ - to store the state of Double Ratchet algorit
2. Protocol commands and events may be changed (even if at the cost of slightly reducing message size) can fit the hash of the ratchet state (32 bytes sha256 would be sufficient), so that the client can determine whether it has the most recent ratchet state or if it needs to retrieve the latest copy. Message size reduction won't affect the users because we use compression, and there is a substantial reserve.
3. Client commands that modify ratchet state would include the hash of the previous ratchet state so that the server can reject or ignore the command in case the previous ratchet state is different or in case command is repeated in case of lost response).
3. Client commands that modify ratchet state would include the hash of the previous ratchet state so that the router can reject or ignore the command in case the previous ratchet state is different or in case command is repeated in case of lost response).
4. The client does not need to retrieve message state for each encryption and decryption operation - it can "speculatively" use the ratchet state it has, and receive correct ratchet state in the "error" response after attempting encryption based on incorrect ratchet state.
## Proposed protocol design
Ratchet state will be stored on the same server that stores message queue, as part of message queue record. 8kb is a sufficient size for this blob (the actual max size is 7800 bytes). The server would also store the hashes of the current and, possibly, the previous ratchet states (TBC).
Ratchet state will be stored on the same router that stores message queue, as part of message queue record. 8kb is a sufficient size for this blob (the actual max size is 7800 bytes). The router would also store the hashes of the current and, possibly, the previous ratchet states (TBC).
While ratchet is used for duplex connection, the connection still has primary queue, and with redundancy the same ratchet state can be stored on all secondary queues.
Ratchet state will be encrypted using secret_box - a symmetric encryption scheme, so PQ-resistant. If ratchet state is stored on more than one server, it has to be encrypted with a different key for each server.
Ratchet state will be encrypted using secret_box - a symmetric encryption scheme, so PQ-resistant. If ratchet state is stored on more than one router, it has to be encrypted with a different key for each router.
Questions: how to rotate the key used to store ratchet? Should key used to encrypt ratchet rotate at the same time when queue is rotated? The latter is a logical option, as it prevents additional complexity and solves the problem anyway. A possible option is to have "ratchet version" that will be used to advance the key used to encrypt ratchet via HKDF.
Security considerations: the scheme may reduce break-in recovery to the points queues are rotated, unless there is some randomness mixed-in into the key derivation (the key used to encrypt ratchet state). But including randomness would defeat the purpose, as other devices wouldn't be able to access the ratchets. Another approach would be to have each device use its own key for encryption, and encrypt to all keys of all devices (or to encrypt key, to avoid size increase). Having multiple encryptions would show how many devices use the queue, but servers already can observe it, so it is a better tradeoff. Another idea would be to rotate the key used to authorize queue commands - we already support multiple recipient keys, and it can be used for multi-device scenario. That would partially mitigate break-in attacks as the attacker who obtained the key from ratchet state would be able to decrypt it, but won't be able to decrypt it (the attacker collusion with the server is not mitigated). Yet another idea would be for each party (device) to share its private (or encapsulation) key and to have a symmetric key (used to encrypt the ratchet state) encrypted (encapsulated) separately for each device. This would reduce the size of the stored data to `ratchet size` + `encrypted key size` * N, so even in case of PQ encryption (e.g. sntrup) the size required to store the ratchet would be under transport block size, while limiting it to say 4-8 devices, which is sufficient.
Security considerations: the scheme may reduce break-in recovery to the points queues are rotated, unless there is some randomness mixed-in into the key derivation (the key used to encrypt ratchet state). But including randomness would defeat the purpose, as other devices wouldn't be able to access the ratchets. Another approach would be to have each device use its own key for encryption, and encrypt to all keys of all devices (or to encrypt key, to avoid size increase). Having multiple encryptions would show how many devices use the queue, but routers already can observe it, so it is a better tradeoff. Another idea would be to rotate the key used to authorize queue commands - we already support multiple recipient keys, and it can be used for multi-device scenario. That would partially mitigate break-in attacks as the attacker who obtained the key from ratchet state would be able to decrypt it, but won't be able to decrypt it (the attacker collusion with the router is not mitigated). Yet another idea would be for each party (device) to share its private (or encapsulation) key and to have a symmetric key (used to encrypt the ratchet state) encrypted (encapsulated) separately for each device. This would reduce the size of the stored data to `ratchet size` + `encrypted key size` * N, so even in case of PQ encryption (e.g. sntrup) the size required to store the ratchet would be under transport block size, while limiting it to say 4-8 devices, which is sufficient.
To participate in multi-device scheme the devices would join the usual group that will be used to share public (encapsulation) device keys and to communicate updates to conversations that were received by the currently "active" device. "Active" means the device that received or sent and processed the message, and while only one device can receive messages from a given queue, device "active" state may be determined per queue, allowing concurrent usage.
@@ -50,7 +50,7 @@ The scheme must be resilient to state updates being lost, and in case of direct
`rsi` - ratchet state on device `i`.
`enc(rs)` - current authoritative ratchet state on the server.
`enc(rs)` - current authoritative ratchet state on the router.
`pt` and `ct` - plaintext and ciphertext messages.
@@ -58,13 +58,13 @@ Encryption is a state transition function ratchetEnc: `(ct, rs') = ratchetEnc(pt
1. Device encrypts the message using the stored ratchet state: `(ct, rsi') = ratchetEnc(pt, rsi)`
2. Device sends modified encrypted ratchet state and the hash of the previous encrypted state to the server that stores the queue: `RSET (hash(enc(rsi)), enc(rsi'))`.
2. Device sends modified encrypted ratchet state and the hash of the previous encrypted state to the router that stores the queue: `RSET (hash(enc(rsi)), enc(rsi'))`.
3. If the hash of the previous state matches state stored on the server (`hash(enc(rsi)) == hash(enc(rs))`), the server updates the state and responds with `ratchet_ok` (that may include the current state or it's hash, for validation). If the hash is different, the server responds with `bad_ratchet(enc(rs))` message that includes the correct ratchet state. These updates must be atomic. In this case device has to update the local ratchet state (provided it can decrypt it), and repeat encryption attempt. If device cannot decrypt the provided ratchet state, it means that the connection is disrupted (possibly, device is removed from device group, but missed the notifications).
3. If the hash of the previous state matches state stored on the router (`hash(enc(rsi)) == hash(enc(rs))`), the router updates the state and responds with `ratchet_ok` (that may include the current state or it's hash, for validation). If the hash is different, the router responds with `bad_ratchet(enc(rs))` message that includes the correct ratchet state. These updates must be atomic. In this case device has to update the local ratchet state (provided it can decrypt it), and repeat encryption attempt. If device cannot decrypt the provided ratchet state, it means that the connection is disrupted (possibly, device is removed from device group, but missed the notifications).
4. After successful state update in primary receiving queue, the device would update it in secondary receiving queues.
5. Device sends encrypted message as usual, via proxy that must be different both from the server that stores the ratchet and from the destination server.
5. Device sends encrypted message as usual, via proxy that must be different both from the router that stores the ratchet and from the destination router.
6. Device broadcasts sent message and new ratchet state to other devices in the device group.
@@ -74,17 +74,17 @@ This protocol is simple, and it minimizes requests when sending the message to o
Decryption is also a state transition function: `(pt, rs') = ratchetDec(ct, rs)`
1. Server sends the message to the device (can be in response to SUB or ACK commands, or with active subscription). Pushed message would include the hash of the currently stored ratchet state: `hash(enc(rs))`.
1. Router sends the message to the device (can be in response to SUB or ACK commands, or with active subscription). Pushed message would include the hash of the currently stored ratchet state: `hash(enc(rs))`.
2. If device has the ratchet state with the same hash (`hash(enc(rs)) == hash(enc(rsi))`), it decrypts the message: `(pt, rsi') = ratchetDec(ct, rsi)`.
3. If device has ratchet state with a different hash, it requests ratchet from the server with additional protocol command `RGET` with response `RCHT (enc(rs))` and updates the local state.
3. If device has ratchet state with a different hash, it requests ratchet from the router with additional protocol command `RGET` with response `RCHT (enc(rs))` and updates the local state.
4. Device decrypts the message `(pt, rsi') = ratchetDec(ct, rsi)` and processes it as usual.
5. Device sends acknowledgement to the server as usual, but now it includes the new ratchet state and the hash of the previous state: `ACK msgId (hash(enc(rsi)), enc(rsi'))`
5. Device sends acknowledgement to the router as usual, but now it includes the new ratchet state and the hash of the previous state: `ACK msgId (hash(enc(rsi)), enc(rsi'))`
6. The server compares ratchet state with stored state hash, and in case it matches it processes `ACK` and responds with `OK` as usual (or `NO_MSG` in case msgId is incorrect, also as usual - it would happen in repeated ACK requests). If ratchet state hash does not match, the server would respond with `bad_ratchet(enc(rs))` - which means that the message was already processed by another device and ratchet was advanced. This is a complex scenario, as the client has to either revert the change from message processing or somehow combine the change with the updates communicated via device group (as a side note, device group can simply re-broadcast messages, not state updates, but it will result in state divergence between devices when different messages are lost).
6. The router compares ratchet state with stored state hash, and in case it matches it processes `ACK` and responds with `OK` as usual (or `NO_MSG` in case msgId is incorrect, also as usual - it would happen in repeated ACK requests). If ratchet state hash does not match, the router would respond with `bad_ratchet(enc(rs))` - which means that the message was already processed by another device and ratchet was advanced. This is a complex scenario, as the client has to either revert the change from message processing or somehow combine the change with the updates communicated via device group (as a side note, device group can simply re-broadcast messages, not state updates, but it will result in state divergence between devices when different messages are lost).
Unlike sending messages, this flow does not require any additional requests in most cases, only requiring requesting message state reconciliation when the same message was received and processed by more than one client, but it does not require re-acknowledgement.
+101
View File
@@ -0,0 +1,101 @@
# Detecting and fixing state with service subscriptions
## Problem
While service certificates and subscriptions hugely decrease startup time and delivery delays on router restarts, they introduce the risk of losing subscriptions in case of state drifts. They also do not provide efficient mechanism for validating that the list of subscribed queues is in sync.
How can the state drift happen?
There are several possibilities:
- lost broker response would make the broker consider that the queue is associated, but the client won't know it, and will have to re-associate. While in itself it is not a problem, as it'll be resolved, it would make drift detected more frequently (regardless of the detection logic used). That service certificates are used on clients with good connection would make it less likely though.
- router state restored from the backup, in case of some failure. Nothing can be done to recover lost queues, but we may restore lost service associations.
- queue blocking or removal by router operator because of policy violation.
- router downgrade (when it loses all service associations) with subsequent upgrade - the client would think queues are associated, while they are not, and won't receive any messages at all in this scenario.
- any other router-side error or logic error.
In addition to the possibility of the drift, we simply need to have confidence that service subscriptions work as intended, without skipping queues. We ignored this consideration for notifications, as the tolerance to lost notifications is higher, but we can't ignore it for messages.
## Solution
Previously considered approach of sending NIL to all queues without messages is very expensive for traffic (most queues don't have messages), and it is also very expensive to detect and validate drift in the client because of asynchronous / concurrent events.
We cannot read all queues into memory, and we cannot aggregate all responses in memory, and we cannot create database writes on every single service subscription to say 1m queues (a realistic number), as it simply won't work well even at the current scale.
An approach of having an efficient way to detect drift, but load the full list of IDs when drift is detected, also won't work well, as drifts may be common, so we need both efficient way to detect there is diff and also to reconcile it.
### Drift detection
Both client and router would maintain the number of associated queues and the "symmetric" hash over the set of queue IDs. The requirements for this hash algorithm are:
- not cryptographically strong, to be fast.
- 128 bits to minimize collisions over the large set of millions of queues.
- symmetric - the result should not depend on ID order.
- allows fast additions and removals.
In this way, every time association is added or removed (including queue marked as deleted), both peers would recompute this hash in the same transaction.
The client would suspend sending and processing any other commands on the router and the queues of this router until SOKS response is received from this router, to prevent drift. It can be achieved with per-router semaphores/locks in memory. UI clients need to become responsive sooner than these responses are received, but we do not service certificates on UI clients, and chat relays may prevent operations on router queues until SOKS response is received.
SOKS response would include both the count of associated queues (as now) and the hash over all associated queue IDs (to be added). If both count and hash match, the client will not do anything. If either does not match the client would perform full sync (see below).
There is a value from doing the same in notification router as well to detect and "fix" drifts.
The algorithm to compute hashes can be the following.
1. Compute hash of each queue ID using xxHash3_128 ([xxhash-ffi](https://hackage.haskell.org/package/xxhash-ffi) library). They don't need to be stored or loaded at once, initially, it can be done with streaming if it is detected on start that there is no pre-computed hash.
2. Combine hashes using XOR. XOR is both commutative and associative, so it would produce the same aggregate hash irrespective of the ID order.
3. Adding queue ID to pre-computed hash requires a single XOR with ID hash: `new_aggregate = aggregate XOR hash(queue_id)`.
4. Removing queue ID from pre-computed hash also requires the same XOR (XOR is involutory, it undoes itself): `new_aggregate = aggregate XOR hash(queue_id)`.
These hashes need to be computed per user/router in the client and per service certificate in the router - on startup both have to validate and compute them once if necessary.
There can be also a start-up option to recompute hashe(s) to detect and fix any errors.
This is all rather simple and would help detecting drifts.
### Synchronization when drift is detected
The assumption here is that in most cases drifts are rare, and isolated to few IDs (e.g., this is the case with notification router).
But the algorithm should be resilient to losing all associations, and it should not be substantially worse than simply restoring all associations or loading all IDs.
We have `c_n` and `c_hash` for client-side count and hash of queue IDs and `s_n` and `s_hash` for router-side, which are returned in SOKS response to SUBS command.
1. If `c_n /= s_n || c_hash /= s_hash`, the client must perform sync.
2. If `abs(c_n - s_n) / max(c_n, s_n) > 0.5`, the client will request the full list of queues (more than half of the queues are different), and will perform diff with the queues it has. While performing the diff the client will continue block operations with this user/router.
3. Otherwise would perform some algorithm for determining the difference between queue IDs between client and router. This algorithm can be made efficient (`O(log N)`) by relying on efficient sorting of IDs and database loading of ranges, via computing and communicating hashes of ranges, and performing a binary search on ranges, with batching to optimize network traffic.
This algorithm is similar to Merkle tree reconcilliation, but it is optimized for database reading of ordered ranges, and for our 16kb block size to minimize network requests.
The algorithm:
1. The client would request all ranges from the router.
2. The router would compute hashes for N ranges of IDs and send them to the client. Each range would include start_id, optional end_id (for single ID ranges) and XOR-hash of the range. N is determined based on the block size and the range size.
3. The client would perform the same computation for the same ranges, and compare them with the returned ranges from the router, while detecting any gaps between ranges and missing range boundaries.
4. If more than half of the ranges don't match, the client would request the full list. Otherwise it would repeat the same algorithm for each mismatched range and for gaps.
It can be further optimized by merging adjacent ranges and by batching all range requests, it is quite simple.
Once the client determines the list of missing and extra queues it can:
- create associations (via SUB) for missing queues,
- request removal of association (a new command, e.g. BUS) for extra queues on the router.
The pseudocode for the algorightm:
For the router to return all ranges or subranges of requested range:
```haskell
getSubRanges :: Maybe (RecipientId, RecipientId) -> [(RecipientId, Maybe RecipientId, Hash)]
getSubRanges range_ = do
((min_id, max_id), s_n) <- case range_ of
Nothing -> getAssociatedQueueRange -- with the certificate in the client session.
Just range -> (range,) <$> getAssociatedQueueCount range
if
| s_n <= max_N -> reply_with_single_queue_ranges
| otherwise -> do
let range_size = s_n `div` max_N
read_all_ranges -- in a recursive loop, with max_id, range_hash and next_min_id in each step
reply_ranges
```
We don't need to implement this synchronization logic right now, so not including client logic here, it's sufficient to implement drift detection, and the action to fix the drift would be to disable and to re-enable certificates via some command-line parameter of CLI.
@@ -0,0 +1,90 @@
# Subscription performance
No protocol changes. This is an implementation RFC addressing subscription performance bottlenecks in both the SMP router and the agent.
## Problem
Subscribing large numbers of queues is slow. A messaging client with ~300K queues per router across 3 routers takes over 1 hour to subscribe. For comparison, the NTF server with ~1M queues per router across 12 routers took 20-30 minutes (prior to NTF client services, now in master).
Even on fast networks (cloud VMs), a client with 1.1M active subscriptions needed ~1.5M attempts (commands sent) to fully subscribe - ~36% retry rate caused by the timeout cascade described below.
### Root causes
#### 1. Router: per-command processing in batches
Batch verification and queue lookups are already done efficiently for the whole batch in `Server.hs`. But `processCommand` is called per-command in a loop - each SUB does its own individual DB query for message peek/delivery. With ~135 SUBs per batch (current SMP version), that's 135 individual DB queries per batch instead of 1 batched query.
For 300K queues, that's ~2200 batches x 135 queries = ~300K individual DB queries on the router, which is the dominant bottleneck when using PostgreSQL storage.
NSUB is cheaper because it just registers for notifications without message delivery - no per-queue DB query.
#### 2. Agent: all queues read and sent at once
`getUserServerRcvQueueSubs` reads all queues for a `(userId, server)` pair in one query with no LIMIT. For 300K queues, the entire result set is loaded into memory, then all ~2200 batches are queued to send without waiting for responses.
The NTF server agent uses cursor-style reading with configurable batch sizes (900 subs per chunk, 90K per DB fetch) and waits for each chunk to be processed before fetching the next.
#### 3. No backpressure on sends
`nonBlockingWriteTBQueue` bypasses the `sndQ` bound by forking a thread when the queue is full. All batches are queued immediately, and all their response timers start simultaneously. A 30-second per-response timeout means later batches time out not because the router is slow to respond to them specifically, but because they're waiting in the router's receive queue behind thousands of earlier commands.
This causes cascading timeouts: timed-out responses trigger `resubscribeSMPSession`, which retries all pending subs. Three consecutive timeouts can trigger connection drop via the monitor thread, causing a full reconnection and retry of everything.
## Solution
### Part 1: Router - batched command processing
Move the per-command processing loop inside command handlers so that commands of the same type within a batch can be processed together.
Current flow:
```
receive batch -> verify all -> lookup queues all -> for each command: processCommand (individual DB query)
```
Proposed flow:
```
receive batch -> verify all -> lookup queues all -> group by command type -> process group:
SUB group: one batched message peek query for all queues
NSUB group: batch registration (already cheap, but can batch DB writes)
other commands: process individually as before
```
For SUB, the batched processing would:
1. Collect all queue IDs from the SUB group
2. Perform a single DB query to peek messages for all queues
3. Distribute results back to individual responses
This reduces ~135 DB queries per batch to 1, cutting router-side DB load by ~100x for subscriptions.
Commands where batching doesn't matter (SEND, ACK, KEY, etc.) continue to be processed individually.
### Part 2: Agent - cursor-based subscription with backpressure
Replace the all-at-once fetch-and-send pattern with cursor-style batching, similar to what the NTF server agent does.
Changes to `subscribeUserServer`:
1. Fetch queues in fixed-size batches (e.g., configurable, default ~1000) using LIMIT/OFFSET or cursor-based pagination.
2. Send each batch and wait for responses before sending the next.
3. Remove the use of `nonBlockingWriteTBQueue` for subscription batches - use blocking writes or structured backpressure so response timers don't start until the batch is actually sent.
This ensures:
- Memory usage is bounded (not 300K queue records in memory at once)
- Response timeouts are meaningful (timer starts when the router receives the batch, not when it's queued locally)
- Retries are scoped to the failed batch, not all pending subs
- Works on slow/lossy networks by naturally pacing sends
### Part 3: Response timeout for batches
The current per-response 30-second timeout doesn't account for batch processing time. Options:
1. **Stagger deadlines**: later responses in a batch get proportionally more time. The `rcvConcurrency` field was designed for this but is never used.
2. **Per-batch timeout**: instead of timing individual responses, timeout the entire batch with a budget proportional to batch size.
3. **No timeout for subscription responses**: since subscriptions are sent as batches with backpressure (Part 2), and the connection is monitored by pings, individual response timeouts may not be needed. A subscription that doesn't get a response will be retried on reconnect.
## Priority and ordering
Part 1 (router batching) gives the biggest improvement and is independent of Parts 2/3.
Part 2 (agent cursor + backpressure) eliminates the retry cascade and is critical for slow networks.
Part 3 (timeout handling) is a refinement that can be addressed after Parts 1 and 2.
+4 -4
View File
@@ -143,8 +143,8 @@ As more protocols are designated as Core IP, development naturally transitions t
| Location | Contents | Count |
|----------|----------|-------|
| `protocol/` | Consolidated specs (SMP v9, Agent v5, XFTP v2, XRCP v1, Push v2, PQDR v1) | 6 specs + overview |
| `rfcs/` root | Active draft proposals | 19 |
| `rfcs/done/` | Implemented, not yet verified | 25 |
| `rfcs/standard/` | Verified against implementation | (to be populated) |
| `protocol/` | Consolidated specs (SMP v19, Agent v7, XFTP v3, XRCP v1, NTF v3, PQDR v1) | 6 specs + overview |
| `rfcs/` root | Active draft proposals | 10 |
| `rfcs/done/` | Implemented, not yet verified | 1 (+10 sub-RFCs) |
| `rfcs/standard/` | Verified against implementation | 31 |
| `rfcs/rejected/` | Draft proposals not accepted | 7 |
@@ -1,15 +1,16 @@
# Send File Page — Web-based XFTP File Transfer
## 1. Problem & Business Case
There is no way to send or receive files using SimpleX without installing the app. A static web page that implements the XFTP protocol client-side would allow anyone with a browser to upload and download files via XFTP servers, promoting app adoption.
There is no way to send or receive files using SimpleX without installing the app. A static web page that implements the XFTP protocol client-side would allow anyone with a browser to upload and download files via XFTP routers, promoting app adoption.
**Business constraints:**
- Web page allows up to 100 MB uploads; app allows up to 1 GB.
- Page must promote app installation (e.g., banner, messaging around limits).
**Security constraint:**
- The server hosting the page must never access file content or file descriptions. The file description is carried in the URL hash fragment (`#`), which browsers do not send to the server.
- The router hosting the page must never access file content or file descriptions. The file description is carried in the URL hash fragment (`#`), which browsers do not send to the router.
- The only way to compromise transfer security is page substitution (serving malicious JS). Mitigations: standard web security (HTTPS, CSP, SRI) and IPFS hosting with page fingerprints published in multiple independent locations.
## 2. Design Overview
@@ -29,7 +30,7 @@ There is no way to send or receive files using SimpleX without installing the ap
│ fetch() over HTTP/2 │ fetch() over HTTP/2
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ XFTP Server 1 │ │ XFTP Server 2 │
│ XFTP Router 1 │ │ XFTP Router 2 │
│ (SNI→web cert) │ │ (SNI→web cert) │
│ (+CORS headers) │ │ (+CORS headers) │
└─────────────────┘ └─────────────────┘
@@ -59,7 +60,7 @@ There is no way to send or receive files using SimpleX without installing the ap
### 3.3 Error States
- File too large (> 100 MB): Show limit message with app install CTA.
- Server unreachable: Retry with exponential backoff, show error after exhausting retries.
- Router unreachable: Retry with exponential backoff, show error after exhausting retries.
- File expired: "This file is no longer available" message.
- Decryption failure: "File corrupted or link invalid" message.
@@ -71,7 +72,7 @@ There is no way to send or receive files using SimpleX without installing the ap
https://example.com/file/#<compressed-base64url-encoded-file-description>
```
- Hash fragment is never sent to the server.
- Hash fragment is never sent to the router.
- Compression: DEFLATE (raw, no gzip/zlib wrapper) — better ratio than LZW for structured text like YAML.
- Encoding: Base64url (RFC 4648 §5) — no `+`, `/`, `=`, or `%` characters.
@@ -79,18 +80,18 @@ Alternative: LZW + base64url if DEFLATE proves problematic. Both should be evalu
### 4.2 Redirect Mechanism
For files with many chunks, the YAML file description can exceed a practical URL length. The threshold is ~600 bytes of compressed+encoded description (configurable).
For files with many data packets, the YAML file description can exceed a practical URL length. The threshold is ~600 bytes of compressed+encoded description (configurable).
**Flow when description is too large:**
1. Serialize recipient file description to YAML.
2. Encrypt YAML using fresh key + nonce (same XSalsa20-Poly1305 as files).
3. Upload encrypted YAML as a single-chunk "file" to one randomly chosen XFTP server.
3. Upload encrypted YAML as a single-packet "file" to one randomly chosen XFTP router.
4. Create redirect description pointing to this uploaded description.
5. Encode redirect description into URL (always small — single chunk).
5. Encode redirect description into URL (always small — single data packet).
**Download with redirect:**
1. Parse URL → redirect description (has `redirect` field with `size` and `digest`).
2. Download the description "file" using the single chunk reference.
2. Download the description "file" using the single data packet reference.
3. Decrypt → get full YAML description.
4. Validate size and digest match redirect metadata.
5. Proceed with normal download using full description.
@@ -99,11 +100,11 @@ For files with many chunks, the YAML file description can exceed a practical URL
These estimates are preliminary and may be incorrect.
| Scenario | Chunks | Compressed+encoded size | URL length |
| Scenario | Data packets | Compressed+encoded size | URL length |
|----------|--------|------------------------|------------|
| Small file (1 chunk, 1 server) | 1 | ~300 bytes | ~350 chars |
| Medium file (5 chunks, 1 server) | 5 | ~500 bytes | ~550 chars |
| Large file (25+ chunks) | 25 | Exceeds threshold → redirect | ~350 chars |
| Small file (1 data packet, 1 router) | 1 | ~300 bytes | ~350 chars |
| Medium file (5 data packets, 1 router) | 5 | ~500 bytes | ~550 chars |
| Large file (25+ data packets) | 25 | Exceeds threshold → redirect | ~350 chars |
## 5. TypeScript XFTP Client Library
@@ -141,7 +142,7 @@ The XFTP wire format uses a custom binary encoding (from `Simplex.Messaging.Enco
- Fields separated by space (0x20).
- `signature`: Ed25519 signature over `(sessionId ++ corrId ++ entityId ++ encodedCommand)`.
- `corrId`: Correlation ID (arbitrary, echoed in response).
- `entityId`: File/chunk ID on server.
- `entityId`: File/data packet ID on router.
- Command: tag + space-separated fields.
- **Padding:** 2-byte big-endian length prefix + message + `#` (0x23) fill to block size (16384 bytes).
@@ -154,7 +155,7 @@ The XFTP wire format uses a custom binary encoding (from `Simplex.Messaging.Enco
| Transit decryption (download) | XSalsa20-Poly1305 (streaming: `cbInit` + `sbDecryptChunk`) | DH shared secret | 24 B | 16 B | libsodium.js |
| Command signing | Ed25519 | 64 B (private) | — | 64 B (sig) | libsodium.js |
| DH key exchange | X25519 | 32 B | — | — | libsodium.js |
| Chunk digest | SHA-256 | — | — | 32 B | Web Crypto API |
| Data packet digest | SHA-256 | — | — | 32 B | Web Crypto API |
| File digest | SHA-512 | — | — | 64 B | Web Crypto API |
| Random bytes | ChaCha20-DRBG | — | — | — | libsodium.js `randombytes_buf` |
@@ -204,7 +205,7 @@ async function sendXFTPCommand(
- Firefox 102+: Supported
- Safari 16.4+: Supported
For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memory).
For older browsers, fall back to `ArrayBuffer` body (buffer entire data packet in memory).
### 5.5 Upload Orchestration
@@ -220,22 +221,22 @@ For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memo
d. Encrypt `'#'` padding in 65536-byte chunks to fill `encSize - authTagSize - fileSize' - 8`
e. Finalize: `sbAuth(state)` → append 16-byte auth tag
6. Compute SHA-512 digest of encrypted data
7. Split into chunks using prepareChunkSizes algorithm:
- > 75% of 4MB → 4MB chunks
- > 75% of 1MB → 1MB + 4MB chunks
- Otherwise → 64KB + 256KB chunks
8. For each chunk (parallel, up to 8 concurrent):
7. Split into data packets using prepareChunkSizes algorithm:
- > 75% of 4MB → 4MB data packets
- > 75% of 1MB → 1MB + 4MB data packets
- Otherwise → 64KB + 256KB data packets
8. For each data packet (parallel, up to 8 concurrent):
a. Generate Ed25519 sender keypair
b. Generate Ed25519 recipient keypair (1 recipient for web)
c. Compute SHA-256 chunk digest
d. Connect to XFTP server (handshake if new connection)
c. Compute SHA-256 data packet digest
d. Connect to XFTP router (handshake if new connection)
e. Send FNEW { sndKey, size, digest } + recipient keys → receive (senderId, [recipientId])
f. Send FPUT with chunk data → receive OK
f. Send FPUT with data packet content → receive OK
g. Report progress
9. Build FileDescription YAML from all chunk metadata
9. Build FileDescription YAML from all data packet metadata
10. If YAML size (compressed+encoded) > threshold:
a. Encrypt YAML as a file
b. Upload encrypted YAML (single chunk) → get redirect description
b. Upload encrypted YAML (single data packet) → get redirect description
c. Use redirect description for URL
11. Compress + base64url encode description
12. Display URL: https://example.com/file/#<encoded>
@@ -247,32 +248,32 @@ For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memo
1. Parse URL hash fragment
2. Base64url decode + decompress → YAML
3. Parse YAML → FileDescription
4. Validate description (sequential chunks, sizes match)
4. Validate description (sequential data packets, sizes match)
5. If redirect field present:
a. Download redirect file (single chunk)
a. Download redirect file (single data packet)
b. Decrypt, validate size+digest, parse inner description
c. Continue with inner description
6. For each chunk (parallel, up to 8 concurrent):
6. For each data packet (parallel, up to 8 concurrent):
a. Generate ephemeral X25519 keypair
b. Connect to XFTP server (web handshake)
b. Connect to XFTP router (web handshake)
c. Send FGET { recipientDhPubKey } → receive (serverDhPubKey, cbNonce) + encrypted body
d. Compute DH shared secret
e. Transit-decrypt chunk body (XSalsa20-Poly1305 with DH secret)
f. Verify chunk digest (SHA-256)
e. Transit-decrypt data packet body (XSalsa20-Poly1305 with DH secret)
f. Verify data packet digest (SHA-256)
g. Send FACK → receive OK
h. Report progress
7. Concatenate all transit-decrypted chunks (in order) → encrypted file
7. Concatenate all transit-decrypted data packets (in order) → encrypted file
8. Verify file digest (SHA-512)
9. File-decrypt entire stream (XSalsa20-Poly1305 with file key + nonce)
10. Extract FileHeader → get original fileName
11. Trigger browser download (Blob + <a download> or File System Access API)
```
## 6. XFTP Server Changes
## 6. XFTP Router Changes
### 6.1 SNI-Based Certificate Switching
The SMP server already implements SNI-based certificate switching (see `Transport/Server.hs:255-269`). The same mechanism must be added to the XFTP server.
The SMP router already implements SNI-based certificate switching (see `Transport/Server.hs:255-269`). The same mechanism must be added to the XFTP router.
**Current SMP implementation:**
```haskell
@@ -292,14 +293,14 @@ T.onServerNameIndication = case sniCredential of
**Certificate setup:**
- XFTP identity certificate: Existing self-signed CA chain (used for protocol identity via fingerprint).
- Web certificate: Standard CA-issued TLS certificate (e.g., Let's Encrypt) for the server's FQDN.
- Web certificate: Standard CA-issued TLS certificate (e.g., Let's Encrypt) for the router's FQDN.
- Both certificates served on the same port (443).
### 6.2 CORS Support
Browsers enforce same-origin policy. The web page (served from `example.com`) must make cross-origin requests to XFTP servers (`xftp1.simplex.im`, etc.).
Browsers enforce same-origin policy. The web page (served from `example.com`) must make cross-origin requests to XFTP routers (`xftp1.simplex.im`, etc.).
**Required server changes:**
**Required router changes:**
1. **Handle OPTIONS preflight requests:**
```
@@ -319,45 +320,45 @@ Browsers enforce same-origin policy. The web page (served from `example.com`) mu
Access-Control-Expose-Headers: *
```
3. **Implementation location:** In `runHTTP2Server` handler or a wrapper around the XFTP request handler. Detect the `Origin` header → add CORS headers. This can be conditional on web mode being enabled in config.
3. **Implementation location:** In `runHTTP2Server` handler or a wrapper around the XFTP request handler. Detect the `Origin` header → add CORS headers. This can be conditional on web mode being enabled in the router config.
**Security consideration:** `Access-Control-Allow-Origin: *` is safe here because:
- All XFTP commands require Ed25519 authentication (per-chunk keys from file description).
- All XFTP commands require Ed25519 authentication (per-packet keys from file description).
- No cookies or browser credentials are involved.
- File content is end-to-end encrypted.
### 6.3 Web Handshake with Server Identity Proof
### 6.3 Web Handshake with Router Identity Proof
**Both SNI and web handshake are required.** They solve different problems:
1. **SNI certificate switching** is required because browsers reject self-signed certificates. The XFTP identity certificate is self-signed (CA chain with offline root), so the server must present a standard CA-issued web certificate (e.g., Let's Encrypt) when a browser connects. SNI is how the server detects this.
1. **SNI certificate switching** is required because browsers reject self-signed certificates. The XFTP identity certificate is self-signed (CA chain with offline root), so the router must present a standard CA-issued web certificate (e.g., Let's Encrypt) when a browser connects. SNI is how the router detects this.
2. **Web handshake with challenge-response** is required because browsers cannot access the TLS certificate fingerprint or the TLS-unique channel binding (`sessionId`). The native client validates XFTP identity by checking the certificate chain fingerprint against the known `keyHash` and binding it to the TLS session. The browser gets none of this — it only knows TLS succeeded with some CA-issued cert. So the XFTP identity must be proven at the protocol level.
**Standard handshake (unchanged for native clients):**
```
1. Client → empty POST body → Server
2. Server → padded { vRange, sessionId, CertChainPubKey } → Client
3. Client → padded { version, keyHash } → Server
4. Server → empty → Client
1. Client → empty POST body → Router
2. Router → padded { vRange, sessionId, CertChainPubKey } → Client
3. Client → padded { version, keyHash } → Router
4. Router → empty → Client
```
**Web handshake (new, when SNI is detected):**
```
1. Client → padded { challenge: 32 random bytes } → Server
2. Server → padded { vRange, sessionId, CertChainPubKey } (header block)
1. Client → padded { challenge: 32 random bytes } → Router
2. Router → padded { vRange, sessionId, CertChainPubKey } (header block)
+ extended body { fullCertChain, signature(challenge ++ sessionId) } → Client
3. Client validates:
- Certificate chain CA fingerprint matches known keyHash
- Signature over (challenge ++ sessionId) is valid under cert's public key
- This proves: server controls XFTP identity key AND is live (not replay)
4. Client → padded { version, keyHash } → Server
5. Server → empty → Client
- This proves: router controls XFTP identity key AND is live (not replay)
4. Client → padded { version, keyHash } → Router
5. Router → empty → Client
```
**Detection mechanism:** The server detects web clients by the `sniCredUsed` flag (already available from the TLS layer). When SNI is detected, the server expects a challenge in the first POST body (non-empty, unlike standard handshake where it is empty). No marker byte is needed — SNI presence is the discriminator.
**Detection mechanism:** The router detects web clients by the `sniCredUsed` flag (already available from the TLS layer). When SNI is detected, the router expects a challenge in the first POST body (non-empty, unlike standard handshake where it is empty). No marker byte is needed — SNI presence is the discriminator.
**Block size note:** The XFTP block size is 16384 bytes (`Protocol.hs:65`). The XFTP identity certificate chain fits within this block. The signed challenge response is sent as an extended body (streamed after the 16384-byte header block), same mechanism as file chunk data.
**Block size note:** The XFTP block size is 16384 bytes (`Protocol.hs:65`). The XFTP identity certificate chain fits within this block. The signed challenge response is sent as an extended body (streamed after the 16384-byte header block), same mechanism as data packet content.
### 6.4 Protocol Version and Handshake Extension
@@ -373,11 +374,11 @@ The XFTP handshake is binary-encoded via the `Encoding` typeclass (`Transport.hs
### 6.5 Serving the Static Page
The XFTP server can optionally serve the static web page itself (similar to how SMP servers serve info pages). When a browser connects via SNI and sends a GET request (not POST), the server serves the HTML/JS/CSS bundle.
The XFTP router can optionally serve the static web page itself (similar to how SMP routers serve info pages). When a browser connects via SNI and sends a GET request (not POST), the router serves the HTML/JS/CSS bundle.
This can be implemented identically to the SMP server's static page serving (`apps/smp-server/web/Static.hs`), using Warp to handle HTTP requests on the same TLS connection.
This can be implemented identically to the SMP router's static page serving (`apps/smp-server/web/Static.hs`), using Warp to handle HTTP requests on the same TLS connection.
Alternatively, the page is hosted on a separate web server (e.g., `files.simplex.chat`). The XFTP servers only need to handle XFTP protocol requests (POST) with CORS headers.
Alternatively, the page is hosted on a separate web server (e.g., `files.simplex.chat`). The XFTP routers only need to handle XFTP protocol requests (POST) with CORS headers.
## 7. Security Analysis
@@ -386,24 +387,24 @@ Alternatively, the page is hosted on a separate web server (e.g., `files.simplex
| Threat | Mitigation | Residual Risk |
|--------|-----------|---------------|
| Page substitution (malicious JS) | HTTPS, CSP, SRI; IPFS hosting with fingerprints in multiple locations | If web server is compromised and IPFS is not used, all guarantees lost. Fundamental limitation of web-based E2E crypto, mitigated by IPFS. |
| MITM between browser and XFTP server | XFTP identity verification via challenge-response handshake | Attacker can relay traffic (see §7.2) but cannot read file content due to E2E encryption. |
| File description leakage | Hash fragment (`#`) is never sent to server | If browser extension or malware reads URL bar, description is exposed. |
| Server learns file content | File encrypted client-side before upload (XSalsa20-Poly1305) | Server sees encrypted chunks only. |
| MITM between browser and XFTP router | XFTP identity verification via challenge-response handshake | Attacker can relay traffic (see §7.2) but cannot read file content due to E2E encryption. |
| File description leakage | Hash fragment (`#`) is never sent to router | If browser extension or malware reads URL bar, description is exposed. |
| Router learns file content | File encrypted client-side before upload (XSalsa20-Poly1305) | Router sees encrypted data packets only. |
| Traffic analysis | File size visible to network observers | Same as native XFTP client. |
### 7.2 Relay Attack Analysis
An attacker who controls the network could relay all traffic between the browser and the real XFTP server:
An attacker who controls the network could relay all traffic between the browser and the real XFTP router:
1. Browser sends challenge to "attacker's server"
2. Attacker relays to real server
3. Real server signs challenge + sessionId with XFTP identity key
1. Browser sends challenge to "attacker's router"
2. Attacker relays to real router
3. Real router signs challenge + sessionId with XFTP identity key
4. Attacker relays signed response to browser
5. Browser validates ✓ (signature is from the real server)
5. Browser validates ✓ (signature is from the real router)
However, the attacker **cannot read file content** because:
- File encryption key is in the hash fragment (never sent over network)
- Transit encryption uses DH key exchange (FGET) — attacker doesn't have server's DH private key
- Transit encryption uses DH key exchange (FGET) — attacker doesn't have router's DH private key
- The attacker can observe transfer sizes and timing, but this is already visible via traffic analysis
The relay attack is equivalent to a passive network observer, which is the same threat model as native XFTP.
@@ -414,6 +415,7 @@ The relay attack is equivalent to a passive network observer, which is the same
|----------|--------------|------------|
| TLS certificate validation | XFTP identity cert via fingerprint pinning | Web CA cert via browser + XFTP identity via challenge-response |
| Session binding | TLS-unique binds to XFTP identity cert | TLS-unique binds to web cert; challenge binds to XFTP identity |
| Code integrity | Binary signed/distributed via app stores | Served over HTTPS; SRI for subresources; IPFS hosting option; vulnerable to server compromise |
| File encryption | XSalsa20-Poly1305 | Same |
| Transit encryption | DH + XSalsa20-Poly1305 | Same |
@@ -421,8 +423,8 @@ The relay attack is equivalent to a passive network observer, which is the same
### 7.4 Layman Security Summary (Displayed on Page)
The web page should display a brief, non-technical security summary explaining to users:
- Files are encrypted in the browser before upload — the server never sees file contents.
- The file link (URL) contains the decryption key in the hash fragment, which the browser never sends to any server.
- Files are encrypted in the browser before upload — the router never sees file contents.
- The file link (URL) contains the decryption key in the hash fragment, which the browser never sends to any router.
- Only someone with the exact link can download and decrypt the file.
- The main risk is if the web page itself is tampered with (page substitution attack). IPFS hosting mitigates this.
- For maximum security, use the SimpleX app instead.
@@ -445,10 +447,10 @@ The web page should display a brief, non-technical security summary explaining t
- Well-understood, readable, auditable by the community.
- Rich crypto ecosystem (libsodium.js provides all needed NaCl primitives as WASM).
- Direct access to browser APIs (fetch, File, ReadableStream, Blob).
- Testable in Node.js against Haskell XFTP server.
- Testable in Node.js against Haskell XFTP router.
- Small bundle size (~200 KB with libsodium WASM).
**Risk:** Exact byte-level wire compatibility requires careful encoding implementation and thorough testing against the Haskell server.
**Risk:** Exact byte-level wire compatibility requires careful encoding implementation and thorough testing against the Haskell router.
### 8.3 Option 3: C to WASM
@@ -476,14 +478,14 @@ The web page should display a brief, non-technical security summary explaining t
4. Handshake encoding/decoding (protocol/handshake.ts) — 18 tests
5. Identity proof verification (crypto/identity.ts) — 15 tests
6. File descriptions: types, YAML, validation (protocol/description.ts) — 13 tests
7. Chunk sizing: prepareChunkSizes, singleChunkSize, etc. (protocol/chunks.ts) — 4 tests
7. Data packet sizing: prepareChunkSizes, singleChunkSize, etc. (protocol/chunks.ts) — 4 tests
8. Transport crypto: cbAuthenticate/cbVerify, transit encrypt/decrypt (protocol/client.ts) — 10 tests
9. Server address parsing (protocol/address.ts) — 3 tests
9. Router address parsing (protocol/address.ts) — 3 tests
10. Download helpers: DH, transit-decrypt, file-decrypt (download.ts) — 11 tests
### Phase 2: XFTP Server Changes — DONE
### Phase 2: XFTP Router Changes — DONE
**Goal:** XFTP servers support web client connections.
**Goal:** XFTP routers support web client connections.
**Completed** (7 Haskell integration tests passing):
1. SNI certificate switching — `TLSServerCredential` mechanism for XFTP
@@ -493,20 +495,20 @@ The web page should display a brief, non-technical security summary explaining t
### Phase 3: HTTP/2 Client + Agent Orchestration
**Goal:** Complete XFTP client that can upload and download files against a real Haskell XFTP server.
**Goal:** Complete XFTP client that can upload and download files against a real Haskell XFTP router.
1. **`client.ts`** ← `Simplex.FileTransfer.Client` — HTTP/2 client via `fetch()` / `node:http2`: connect + handshake, sendCommand, createChunk, uploadChunk, downloadChunk, deleteChunk, ackChunk, ping.
2. **`agent.ts`** ← `Simplex.FileTransfer.Client.Main` — Upload orchestration (encrypt → chunk → register → upload → build description), download orchestration (parse → download → verify → decrypt → ack), URL encoding with DEFLATE compression (§4.1).
2. **`agent.ts`** ← `Simplex.FileTransfer.Client.Main` — Upload orchestration (encrypt → split into data packets → register → upload → build description), download orchestration (parse → download → verify → decrypt → ack), URL encoding with DEFLATE compression (§4.1).
### Phase 4: Integration Testing
**Goal:** Prove the TypeScript client is wire-compatible with the Haskell server.
**Goal:** Prove the TypeScript client is wire-compatible with the Haskell router.
1. **Test harness** — Haskell-driven tests in `XFTPWebTests.hs` (same pattern as per-function tests).
2. **Upload test** — TypeScript uploads file → Haskell client downloads it → verify contents match.
3. **Download test** — Haskell client uploads file → TypeScript downloads it → verify contents match.
2. **Upload test** — TypeScript uploads file → Haskell client downloads it → verify content matches.
3. **Download test** — Haskell client uploads file → TypeScript downloads it → verify content matches.
4. **Round-trip test** — TypeScript upload → TypeScript download → verify.
5. **Edge cases** — Single chunk, many chunks, exactly-sized chunks, redirect descriptions.
5. **Edge cases** — Single data packet, many data packets, exactly-sized data packets, redirect descriptions.
### Phase 5: Web Page
@@ -517,11 +519,11 @@ The web page should display a brief, non-technical security summary explaining t
3. **Download UI** — Parse URL, show file info, download button, progress circle.
4. **App install CTA** — Banner/messaging promoting SimpleX app for larger files.
### Phase 6: Server-Hosted Page (Optional)
### Phase 6: Router-Hosted Page (Optional)
**Goal:** XFTP servers can optionally serve the web page themselves.
**Goal:** XFTP routers can optionally serve the web page themselves.
1. **Static file serving** — Similar to SMP server's `attachStaticFiles`.
1. **Static file serving** — Similar to SMP router's `attachStaticFiles`.
2. **GET handler** — When web client sends HTTP GET (not POST), serve HTML page.
3. **Page generation** — Embed page bundle at server build time.
@@ -588,9 +590,9 @@ cabal test --ghc-options -O0 --test-option=--match="/XFTP Web Client/"
**Random inputs:** Haskell tests can use QuickCheck to generate random inputs each run, not just hardcoded values. This catches edge cases that fixed test vectors miss.
### 10.2 Integration Tests (TS-driven, spawns Haskell server)
### 10.2 Integration Tests (TS-driven, spawns Haskell router)
**Only attempted after all per-function tests (§10.1) pass.** These are end-to-end tests that verify the full upload/download pipeline works against a real XFTP server.
**Only attempted after all per-function tests (§10.1) pass.** These are end-to-end tests that verify the full upload/download pipeline works against a real XFTP router.
**Approach:** Node.js test (`xftp-web/test/integration.test.ts`) spawns `xftp-server` and `xftp` CLI as subprocesses.
@@ -615,7 +617,7 @@ cabal test --ghc-options -O0 --test-option=--match="/XFTP Web Client/"
3. TypeScript upload + download round-trip.
4. Web handshake with challenge-response validation.
5. Redirect descriptions (large file → compressed description upload).
6. Multiple chunks across multiple servers.
6. Multiple data packets across multiple routers.
7. Error cases: expired file, auth failure, digest mismatch.
### 10.3 Browser Tests
@@ -635,7 +637,7 @@ The per-function tests (§10.1) must pass before attempting integration tests (
5. **Protocol encoding** — command/response encoding, transmission framing (§12.2, §12.3)
6. **Handshake** — handshake type encoding/decoding (§12.9)
7. **Description** — YAML serialization, validation (§12.12–§12.14)
8. **Chunk sizing**`prepareChunkSizes`, `getChunkDigest` (§12.11)
8. **Data packet sizing**`prepareChunkSizes`, `getChunkDigest` (§12.11)
9. **Transport client**`sendCommand`, `createChunk`, `uploadChunk`, `downloadChunk` (§12.10)
10. **Integration** — full upload/download round-trips (§10.2)
@@ -660,7 +662,7 @@ The TypeScript implementation must reimplement the exact streaming logic using l
### 11.3 Web Client Detection
Both SNI and web handshake are mandatory (see §6.3). SNI detection (`sniCredUsed` flag) is the discriminator — when SNI is detected, the server expects the web handshake variant.
Both SNI and web handshake are mandatory (see §6.3). SNI detection (`sniCredUsed` flag) is the discriminator — when SNI is detected, the router expects the web handshake variant.
### 11.4 URL Compression
@@ -677,32 +679,32 @@ XSalsa20-Poly1305 streaming encryption/decryption is sequential — each 64KB bl
**Upload flow:**
1. `File.stream()` → encrypt sequentially (state threading) → buffer encrypted output
2. Compute SHA-512 digest of encrypted data
3. Split into chunks, upload in parallel to 8 randomly selected servers (from 6 default servers in `Presets.hs`)
3. Split into data packets, upload in parallel to 8 randomly selected routers (from 6 default routers in `Presets.hs`)
**Download flow:**
1. Download chunks in parallel from servers → buffer encrypted data
1. Download data packets in parallel from routers → buffer encrypted data
2. Decrypt sequentially (state threading) → verify auth tag
3. Trigger browser save
Both directions buffer ~100 MB of encrypted data. The approach should be symmetric.
**Option A — Memory buffer:** Buffer encrypted data as `ArrayBuffer`. 100 MB peak memory is feasible on modern devices. Simple implementation, no Web Worker needed. Chunk slicing is zero-copy via `ArrayBuffer.slice()`.
**Option A — Memory buffer:** Buffer encrypted data as `ArrayBuffer`. 100 MB peak memory is feasible on modern devices. Simple implementation, no Web Worker needed. Data packet slicing is zero-copy via `ArrayBuffer.slice()`.
**Option B — OPFS ([Origin Private File System](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system)):** Write encrypted data to OPFS instead of holding in memory. OPFS storage quota is shared with IndexedDB/Cache API — typically hundreds of MB to several GB ([quota details](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria)). The fast synchronous API (`createSyncAccessHandle()`) requires a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle) but is [3-4x faster than IndexedDB](https://web.dev/articles/origin-private-file-system). The async API (`createWritable()`) works on the main thread.
**Decision:** Use OPFS with a Web Worker. While 100 MB fits in memory, OPFS future-proofs the implementation for raising the file size limit (250 MB, 500 MB, etc.) without code changes. The Web Worker also keeps the main thread responsive during encryption/decryption. The implementation cost is modest — a single worker that runs the sequential crypto pipeline, reading/writing OPFS files.
### 11.7 Server Page Hosting
### 11.7 Router Page Hosting
Excluded from initial implementation. Added at the very end (Phase 5) as optional feature. Initial deployment serves the page from a separate web host.
Excluded from initial implementation. Added at the very end (Phase 5) as optional feature. Initial deployment serves the page from a separate web server.
### 11.8 File Expiry Communication
Hardcode 48 hours for standalone web page. Server-hosted page can use server-configurable TTL. The page should also display which XFTP servers were used for the upload.
Hardcode 48 hours for standalone web page. Router-hosted page can use router-configurable TTL. The page should also display which XFTP routers were used for the upload.
### 11.9 Concurrent Operations
8 parallel operations in the browser. The Haskell CLI uses 16, but browsers have per-origin connection limits (6-8). Since chunks typically go to different servers (different origins), 8 provides good parallelism without hitting browser limits.
8 parallel operations in the browser. The Haskell CLI uses 16, but browsers have per-origin connection limits (6-8). Since data packets typically go to different routers (different origins), 8 provides good parallelism without hitting browser limits.
## 12. Haskell-to-TypeScript Function Mapping
@@ -861,13 +863,13 @@ Note: `encryptFile` does NOT use `padLazy` or `sbEncryptTailTag`. It manually pr
**`decryptChunks` algorithm** (lines 57-111) — two paths:
**Single chunk (one file, line 60):** Calls `sbDecryptTailTag(key, nonce, encSize - authTagSize, data)` directly. This internally decrypts, verifies auth tag, and strips the 8-byte length prefix + padding via `unPad`. Returns `(authOk, content)`. Then parses `FileHeader` from content.
**Single data packet (one file, line 60):** Calls `sbDecryptTailTag(key, nonce, encSize - authTagSize, data)` directly. This internally decrypts, verifies auth tag, and strips the 8-byte length prefix + padding via `unPad`. Returns `(authOk, content)`. Then parses `FileHeader` from content.
**Multi-chunk (line 67):**
**Multi-packet (line 67):**
1. `sbInit(key, nonce)` → init state
2. Decrypt first chunk file: `sbDecryptChunkLazy(state, chunk)``splitLen` extracts 8-byte `expectedLen` → parse `FileHeader`
3. Decrypt middle chunk files: `sbDecryptChunkLazy(state, chunk)` loop, write to output, accumulate `len`
4. Decrypt last chunk file: split off last 16 bytes as auth tag → `sbDecryptChunkLazy(state, remaining)` → truncate padding using `expectedLen` vs accumulated `len` → verify `sbAuth(finalState) == authTag`
2. Decrypt first data packet: `sbDecryptChunkLazy(state, chunk)``splitLen` extracts 8-byte `expectedLen` → parse `FileHeader`
3. Decrypt middle data packets: `sbDecryptChunkLazy(state, chunk)` loop, write to output, accumulate `len`
4. Decrypt last data packet: split off last 16 bytes as auth tag → `sbDecryptChunkLazy(state, remaining)` → truncate padding using `expectedLen` vs accumulated `len` → verify `sbAuth(finalState) == authTag`
**`FileHeader`** (`Types.hs:35`): `{fileName :: String, fileExtra :: Maybe String}`, parsed via `smpP`.
@@ -888,18 +890,18 @@ XFTP handshake types and encoding.
### 12.10 `protocol/client.ts``Simplex/FileTransfer/Client.hs` (crypto primitives) — DONE
Transport-level crypto for command authentication and chunk encryption/decryption.
Transport-level crypto for command authentication and data packet encryption/decryption.
| TypeScript function | Haskell function | Description | Status |
|---|---|---|---|
| `cbAuthenticate(peerPub, ownPriv, nonce, msg)` | `C.cbAuthenticate` | 80-byte crypto_box authenticator | ✓ |
| `cbVerify(peerPub, ownPriv, nonce, auth, msg)` | `C.cbVerify` | Verify authenticator | ✓ |
| `encryptTransportChunk(dhSecret, nonce, plain)` | `sendEncFile` | Encrypt chunk (tag appended) | ✓ |
| `decryptTransportChunk(dhSecret, nonce, enc)` | `receiveEncFile` | Decrypt chunk (tag verified) | ✓ |
| `encryptTransportChunk(dhSecret, nonce, plain)` | `sendEncFile` | Encrypt data packet (tag appended) | ✓ |
| `decryptTransportChunk(dhSecret, nonce, enc)` | `receiveEncFile` | Decrypt data packet (tag verified) | ✓ |
### 12.11 `protocol/chunks.ts``Simplex/FileTransfer/Chunks.hs` + `Client.hs` — DONE
Chunk size selection and file splitting.
Data packet size selection and file splitting.
| TypeScript function/constant | Haskell equivalent | Status |
|---|---|---|
@@ -944,7 +946,7 @@ HTTP/2 XFTP client using `node:http2` (Node.js) or `fetch()` (browser). Transpil
**XFTPClient state** (returned by `connectXFTP`):
- HTTP/2 session (node: `ClientHttp2Session`, browser: base URL for fetch)
- `thParams`: `{sessionId, blockSize, thVersion, thAuth}` from handshake
- Server address for reconnection
- Router address for reconnection
**sendXFTPCommand wire format:**
1. `xftpEncodeAuthTransmission(thParams, pKey, (corrId, fId, cmd))` → padded 16KB block
@@ -960,16 +962,16 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally
| TypeScript function | Haskell function | Line | Description |
|---|---|---|---|
| `encryptFileForUpload(file, fileName)` | `encryptFileForUpload` | 264 | key/nonce → encrypt → digest → chunk specs |
| `encryptFileForUpload(file, fileName)` | `encryptFileForUpload` | 264 | key/nonce → encrypt → digest → data packet specs |
| `uploadFile(client, chunkSpecs, servers, numRcps)` | `uploadFile` | 285 | Parallel upload (up to 16 concurrent) |
| `uploadFileChunk(client, chunkNo, spec, server)` | `uploadFileChunk` | 301 | FNEW + FPUT for one chunk |
| `uploadFileChunk(client, chunkNo, spec, server)` | `uploadFileChunk` | 301 | FNEW + FPUT for one data packet |
| `createRcvFileDescriptions(fd, sentChunks)` | `createRcvFileDescriptions` | 329 | Build per-recipient descriptions |
| `createSndFileDescription(fd, sentChunks)` | `createSndFileDescription` | 361 | Build sender (deletion) description |
**Upload call sequence** (`cliSendFileOpts`, line 243):
1. `encryptFileForUpload``randomSbKey` + `randomCbNonce``encryptFile``sha512Hash` digest → `prepareChunkSpecs`
2. `uploadFile` — for each chunk: generate sender/recipient key pairs, `createXFTPChunk`, `uploadXFTPChunk`
3. `createRcvFileDescriptions` — assemble `FileDescription` per recipient from sent chunks
2. `uploadFile` — for each data packet: generate sender/recipient key pairs, `createXFTPChunk`, `uploadXFTPChunk`
3. `createRcvFileDescriptions` — assemble `FileDescription` per recipient from sent data packets
4. `createSndFileDescription` — assemble sender description with deletion keys
**Download functions:**
@@ -977,17 +979,17 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally
| TypeScript function | Haskell function | Line | Description |
|---|---|---|---|
| `downloadFile(description)` | `cliReceiveFile` | 388 | Full download: parse → download → verify → decrypt |
| `downloadFileChunk(client, chunk)` | `downloadFileChunk` | 418 | FGET + transit-decrypt one chunk |
| `ackFileChunk(client, chunk)` | `acknowledgeFileChunk` | 440 | FACK one chunk |
| `deleteFile(description)` | `cliDeleteFile` | 455 | FDEL for all chunks |
| `downloadFileChunk(client, chunk)` | `downloadFileChunk` | 418 | FGET + transit-decrypt one data packet |
| `ackFileChunk(client, chunk)` | `acknowledgeFileChunk` | 440 | FACK one data packet |
| `deleteFile(description)` | `cliDeleteFile` | 455 | FDEL for all data packets |
**Download call sequence** (`cliReceiveFile`, line 388):
1. Parse and validate `FileDescription` from YAML
2. Group chunks by server
3. Parallel download: `downloadXFTPChunk` per chunk (up to 16 concurrent)
4. Verify file digest (SHA-512) over concatenated encrypted chunks
2. Group data packets by router
3. Parallel download: `downloadXFTPChunk` per data packet (up to 16 concurrent)
4. Verify file digest (SHA-512) over concatenated encrypted data packets
5. `decryptChunks` — file-level decrypt with auth tag verification
6. Parallel acknowledge: `ackXFTPChunk` per chunk
6. Parallel acknowledge: `ackXFTPChunk` per data packet
**URL encoding (§4.1):**
@@ -1004,7 +1006,7 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally
2. Send `FGET(rcvDhPubKey)` → receive `FRFile(sndDhPubKey, cbNonce)` + encrypted body
3. Compute DH shared secret: `dh'(sndDhPubKey, rcvDhPrivKey)` (`Crypto.hs:1280`)
4. Transit-decrypt body via `receiveSbFile` (`Transport.hs:176`): `cbInit(dhSecret, cbNonce)``sbDecryptChunk` loop (`fileBlockSize` = 16384-byte blocks, `Transport/HTTP2/File.hs:14`) → `sbAuth` tag verification at end
5. Verify chunk digest (SHA-256): `getChunkDigest` (`Client.hs:346`)
5. Verify data packet digest (SHA-256): `getChunkDigest` (`Client.hs:346`)
### 12.18 Per-Function Testing: Haskell Drives Node
@@ -1,12 +1,12 @@
# XFTP Server: SNI, CORS, and Web Support
# XFTP Router: SNI, CORS, and Web Support
Implementation details for Phase 3 of `rfcs/2026-01-30-send-file-page.md` (sections 6.1-6.4).
## 1. Overview
The XFTP server is extended to support web browser clients by:
The XFTP router is extended to support web browser clients by:
1. **SNI-based TLS certificate switching** — Present a CA-issued web certificate (e.g., Let's Encrypt) to browsers, while continuing to present the self-signed XFTP identity certificate to native clients.
1. **SNI-based TLS certificate switching** — Present a CA-issued web certificate (e.g., Let's Encrypt) to browsers, while continuing to present the self-signed XFTP identity certificate to native XFTP clients.
2. **CORS headers** — Add CORS response headers on SNI connections so browsers allow cross-origin XFTP requests.
3. **Configuration**`[WEB]` INI section for HTTPS cert/key paths; opt-in (commented out by default).
@@ -16,11 +16,11 @@ Web handshake (challenge-response identity proof, §6.3 of parent RFC) is not ye
### 2.1 Reusing the SMP Pattern
The SMP server already implements SNI-based certificate switching via `TLSServerCredential` and `runTransportServerState_` (see `rfcs/2024-09-15-shared-port.md`). The XFTP server applies the same pattern with one key difference: both native and web XFTP clients use HTTP/2 transport, whereas SMP switches between raw SMP protocol and HTTP entirely.
The SMP router already implements SNI-based certificate switching via `TLSServerCredential` and `runTransportServerState_` (see `rfcs/2024-09-15-shared-port.md`). The XFTP router applies the same pattern with one key difference: both native and web XFTP clients use HTTP/2 transport, whereas SMP switches between raw SMP protocol and HTTP entirely.
### 2.2 Approach
When `httpServerCreds` is configured, the XFTP server bypasses `runHTTP2Server` and uses `runTransportServerState_` directly to obtain the per-connection `sniUsed` flag. It then sets up HTTP/2 manually on each TLS connection using `withHTTP2` (same internals as `runHTTP2ServerWith_`). The `sniUsed` flag is captured in the closure and shared by all HTTP/2 requests on that connection.
When `httpServerCreds` is configured, the XFTP router bypasses `runHTTP2Server` and uses `runTransportServerState_` directly to obtain the per-connection `sniUsed` flag. It then sets up HTTP/2 manually on each TLS connection using `withHTTP2` (same internals as `runHTTP2ServerWith_`). The `sniUsed` flag is captured in the closure and shared by all HTTP/2 requests on that connection.
When `httpServerCreds` is absent, the existing `runHTTP2Server` path is unchanged.
@@ -33,7 +33,7 @@ Browser client (SNI) ──TLS──> Web CA cert ──HTTP/2──>
The web certificate file (e.g., `web.crt`) must contain the full chain: leaf certificate followed by the signing CA certificate. `loadServerCredential` uses `T.credentialLoadX509Chain` which reads all PEM blocks from the file.
The client validates the chain by comparing `idCert` fingerprint (the CA cert, second in the 2-cert chain) against the known `keyHash`. This is the same validation as for XFTP identity certificates — the CA that signed the web cert must match the XFTP server's identity.
The client validates the chain by comparing `idCert` fingerprint (the CA cert, second in the 2-cert chain) against the known `keyHash`. This is the same validation as for XFTP identity certificates — the CA that signed the web cert must match the XFTP router's identity.
## 3. CORS Support
@@ -69,7 +69,7 @@ Access-Control-Max-Age: 86400
### 3.4 Security
`Access-Control-Allow-Origin: *` is safe because:
- All XFTP commands require Ed25519 authentication (per-chunk keys from file description).
- All XFTP commands require Ed25519 authentication (per-packet keys from file description).
- No cookies or browser credentials are involved.
- File content is end-to-end encrypted.
@@ -87,9 +87,9 @@ Commented out by default — web support is opt-in.
### 4.2 Behavior
- `[WEB]` section not configured: silently ignored, server operates normally for native clients only.
- `[WEB]` section not configured: silently ignored, router operates normally for native clients only.
- `[WEB]` section configured with valid cert/key paths: SNI + CORS enabled.
- `[WEB]` section configured with missing cert files: warning + continue (non-fatal, unlike SMP where it is fatal).
- `[WEB]` section configured with missing cert files: warning + continue (non-fatal, unlike SMP router where it is fatal).
## 5. Files Modified
@@ -146,9 +146,9 @@ Added SNI and CORS tests as a subsection within `xftpServerTests` (6 tests):
3. **CORS headers** — SNI POST request includes `Access-Control-Allow-Origin: *` and `Access-Control-Expose-Headers: *`.
4. **OPTIONS preflight** — SNI OPTIONS request returns all CORS preflight headers.
5. **No CORS without SNI** — Non-SNI POST request has no CORS headers.
6. **File chunk delivery** — Full XFTP file chunk upload/download through SNI-enabled server verifying no regression.
6. **Data packet delivery** — Full XFTP data packet upload/download through SNI-enabled router verifying no regression.
## 6. Remaining Work
- **Web handshake** (§6.3 of parent RFC): Challenge-response identity proof for SNI connections. The server detects web clients via the `sniUsed` flag and expects a 32-byte challenge in the first POST body (non-empty, unlike standard handshake). Response includes full cert chain + signature over `(challenge ++ sessionId)`.
- **Web handshake** (§6.3 of parent RFC): Challenge-response identity proof for SNI connections. The router detects web clients via the `sniUsed` flag and expects a 32-byte challenge in the first POST body (non-empty, unlike standard handshake). Response includes full cert chain + signature over `(challenge ++ sessionId)`.
- **Static page serving** (§6.5 of parent RFC): Optional serving of the web page HTML/JS bundle on GET requests.
@@ -1,6 +1,6 @@
# Web Handshake — Challenge-Response Identity Proof
RFC §6.3: Server proves XFTP identity to web clients independently of TLS CA infrastructure.
RFC §6.3: Router proves XFTP identity to web clients independently of TLS CA infrastructure.
## 1. Protocol
@@ -29,7 +29,7 @@ Server → empty → Client
**Detection**: `sniUsed` per-connection flag. Non-empty hello allowed only when `sniUsed`. Empty hello with SNI → standard handshake.
**Why both steps 3 and 4**: Native clients verify `signedPubKey` using the TLS peer certificate (`serverKey` from `getServerVerifyKey`), which is the XFTP identity cert in non-SNI connections — TLS provides this binding. Web clients cannot access TLS peer certificate data (browser API limitation; TLS presents the web CA cert but provides no API to extract it). So web clients must verify at the application layer using `authPubKey.certChain`, which always contains the XFTP identity chain regardless of which cert TLS used. Step 3 proves the server holds its identity key *right now* (freshness via random challenge). Step 4 proves the DH session key was signed by the identity key holder (prevents MITM key substitution). Together they give web clients some assurance native clients get from TLS, except channel binding for commands.
**Why both steps 3 and 4**: Native clients verify `signedPubKey` using the TLS peer certificate (`serverKey` from `getServerVerifyKey`), which is the XFTP identity cert in non-SNI connections — TLS provides this binding. Web clients cannot access TLS peer certificate data (browser API limitation; TLS presents the web CA cert but provides no API to extract it). So web clients must verify at the application layer using `authPubKey.certChain`, which always contains the XFTP identity chain regardless of which cert TLS used. Step 3 proves the router holds its identity key *right now* (freshness via random challenge). Step 4 proves the DH session key was signed by the identity key holder (prevents MITM key substitution). Together they give web clients some assurance native clients get from TLS, except channel binding for commands.
## 2. Type Changes — `src/Simplex/FileTransfer/Transport.hs`
@@ -56,7 +56,7 @@ Same `Tail compat` pattern as server handshake.
Both types use `(..)` export — new fields auto-exported.
## 3. Server Changes — `src/Simplex/FileTransfer/Server.hs`
## 3. Router Changes — `src/Simplex/FileTransfer/Server.hs`
### `XFTPTransportRequest` (line 88)
@@ -176,7 +176,7 @@ Remove `extractCertEd25519Key` (replaced by generic path). Keep `extractCertPubl
### 10.5 Tests — `tests/XFTPWebTests.hs`
**Integration test**: Switch from `withXFTPServerEd25519SNI` (Ed25519 fixtures) to `withXFTPServerSNI` (default Ed448 fixtures). Update fingerprint source from `tests/fixtures/ed25519/ca.crt` to `tests/fixtures/ca.crt`.
**Integration test**: Switch from `withXFTPServerEd25519SNI` (Ed25519 fixtures) to `withXFTPServerSNI` (default Ed448 fixtures). Update fingerprint source from `tests/fixtures/ed25519/ca.crt` to the default `tests/fixtures/ca.crt`.
Optionally add a second integration test with Ed25519 to cover both paths, or rely on existing unit tests for Ed25519 coverage.
@@ -20,7 +20,7 @@ Build a static web page for browser-based XFTP file transfer (Phase 5 of master
Two build variants:
- **Local**: single test server at `localhost:7000` (development/testing)
- **Production**: 12 preset XFTP servers (6 SimpleX + 6 Flux)
- **Production**: 12 preset XFTP routers (6 SimpleX + 6 Flux)
Uses Vite for bundling (already a dependency via vitest). No CSS framework — plain CSS per RFC spec.
@@ -258,7 +258,7 @@ export function pickRandomServer(servers: XFTPServer[]): XFTPServer {
### 4.3 Assumption
Production XFTP servers must have `[WEB]` section configured with a CA-signed certificate for browser TLS. Without this, browsers will reject the self-signed XFTP identity cert. The local test server uses `tests/fixtures/` certs which Chromium accepts via `ignoreHTTPSErrors`.
Production XFTP routers must have `[WEB]` section configured with a CA-signed certificate for browser TLS. Without this, browsers will reject the self-signed XFTP identity cert. The local test router uses `tests/fixtures/` certs which Chromium accepts via `ignoreHTTPSErrors`.
## 5. Page Structure & UI
@@ -293,7 +293,7 @@ Both upload-complete and download-ready states display a brief non-technical sec
### 5.5 File expiry
Display on upload-complete state: "Files are typically available for 48 hours." This is an approximation — actual expiry depends on each XFTP server's `[STORE_LOG]` retention configuration. The 48-hour figure matches the current preset server defaults.
Display on upload-complete state: "Files are typically available for 48 hours." This is an approximation — actual expiry depends on each XFTP router's `[STORE_LOG]` retention configuration. The 48-hour figure matches the current preset router defaults.
### 5.6 Styling
@@ -19,10 +19,10 @@ This document specifies comprehensive Playwright E2E tests for the XFTP web page
- **Upload flow**: File selection (picker + drag-drop), validation, progress, cancellation, link sharing, error handling
- **Download flow**: Invalid link handling, download button, progress, file save, error states
- **Edge cases**: Boundary file sizes, special characters, network failures, multi-chunk files with redirect, UI information display
- **Edge cases**: Boundary file sizes, special characters, network failures, multi-packet files with redirect, UI information display
**Key constraints**:
- Tests run against a local XFTP server (started via `globalSetup.ts`)
- Tests run against a local XFTP router (started via `globalSetup.ts`)
- Server port is dynamic (read from `/tmp/xftp-test-server.port`)
- Browser uses `--ignore-certificate-errors` for self-signed certs
- OPFS and Web Workers are required (Chromium supports both)
@@ -50,7 +50,7 @@ xftp-web/
### 2.2 Prerequisites
- `globalSetup.ts` starts the XFTP server and writes port to `PORT_FILE`
- `globalSetup.ts` starts the XFTP router and writes port to `PORT_FILE`
- Tests must read the port dynamically: `readFileSync(PORT_FILE, 'utf-8').trim()`
- Vite builds and serves the page at `http://localhost:4173`
@@ -699,7 +699,7 @@ test('concurrent downloads from same link', async ({browser}) => {
})
```
### 6.7 Redirect File Handling (Multi-chunk)
### 6.7 Redirect File Handling (Multi-packet)
**Test ID**: `edge-redirect-file`
@@ -786,7 +786,7 @@ test('download page shows file size and security note', async ({uploadPage, down
### Phase 7: Error Recovery and Advanced (Priority: Low)
22. `upload-error-retry` - Retry after error
23. `edge-concurrent-downloads` - Concurrent access
24. `edge-redirect-file` - Multi-chunk file with redirect (slow)
24. `edge-redirect-file` - Multi-packet file with redirect (slow)
25. `edge-ui-info` - Expiry message, security notes
---
@@ -2,27 +2,27 @@
## 1. Problem Statement
Browser HTTP/2 connection pooling reuses TLS connections across page navigations (same origin = same connection pool). The XFTP server maintains per-TLS-connection session state in `TMap SessionId Handshake` keyed by `tlsUniq tls`. When a browser navigates from the upload page to the download page (or reloads), the new page sends a fresh ClientHello on the reused HTTP/2 connection. The server is already in `HandshakeAccepted` state for that connection, so it routes the request to `processRequest`, which expects a 16384-byte command block but receives a 34-byte ClientHello → `ERR BLOCK`.
Browser HTTP/2 connection pooling reuses TLS connections across page navigations (same origin = same connection pool). The XFTP router maintains per-TLS-connection session state in `TMap SessionId Handshake` keyed by `tlsUniq tls`. When a browser navigates from the upload page to the download page (or reloads), the new page sends a fresh ClientHello on the reused HTTP/2 connection. The server is already in `HandshakeAccepted` state for that connection, so it routes the request to `processRequest`, which expects a 16384-byte command block but receives a 34-byte ClientHello → `ERR BLOCK`.
**Root cause**: The server cannot distinguish a ClientHello from a command on an already-handshaked connection because both arrive on the same HTTP/2 connection (same `tlsUniq`), and there is no content-level discriminator (ClientHello is unpadded, but the server never gets to parse it — the size check in `processRequest` rejects it first).
**Root cause**: The router cannot distinguish a ClientHello from a command on an already-handshaked connection because both arrive on the same HTTP/2 connection (same `tlsUniq`), and there is no content-level discriminator (ClientHello is unpadded, but the router never gets to parse it — the size check in `processRequest` rejects it first).
**Browser limitation**: `fetch()` provides zero control over HTTP/2 connection pooling. There is no browser API to force a new connection or detect connection reuse before a request is sent.
## 2. Solution Summary
Add an HTTP header `xftp-web-hello` to web ClientHello requests. When the server sees this header on an already-handshaked connection (`HandshakeAccepted` state), it re-runs `processHello` **reusing the existing session keys** (same X25519 key pair from the original handshake). The client then completes the normal handshake flow (sends ClientHandshake, receives ack) and proceeds with commands.
Add an HTTP header `xftp-web-hello` to web ClientHello requests. When the router sees this header on an already-handshaked connection (`HandshakeAccepted` state), it re-runs `processHello` **reusing the existing session keys** (same X25519 key pair from the original handshake). The client then completes the normal handshake flow (sends ClientHandshake, receives ack) and proceeds with commands.
Key properties:
- Server reuses existing `serverPrivKey` — no new key material generated on re-handshake, so `thAuth` remains consistent with any in-flight commands on concurrent HTTP/2 streams.
- Router reuses existing `serverPrivKey` — no new key material generated on re-handshake, so `thAuth` remains consistent with any in-flight commands on concurrent HTTP/2 streams.
- Header is only checked when `sniUsed` is true (web/browser connections). Native XFTP clients are unaffected.
- CORS preflight already allows all headers (`Access-Control-Allow-Headers: *`).
- Web clients always send this header on ClientHello — it's harmless on first connection (`Nothing` state) and enables re-handshake on reused connections (`HandshakeAccepted` state).
## 3. Detailed Technical Design
### 3.1 Server change: parameterize `processHello` (`src/Simplex/FileTransfer/Server.hs`)
### 3.1 Router change: parameterize `processHello` (`src/Simplex/FileTransfer/Server.hs`)
The entire server change is parameterizing the existing `processHello` with `Maybe C.PrivateKeyX25519`. Zero new functions.
The entire router change is parameterizing the existing `processHello` with `Maybe C.PrivateKeyX25519`. Zero new functions.
#### Current code (lines 165-191):
@@ -125,7 +125,7 @@ Add optional `headers?` parameter to `Transport.post()`, thread it through `fetc
### 3.5 Haskell test (`tests/XFTPServerTests.hs`)
Add `testWebReHandshake` next to the existing `testWebHandshake` (line 504). It reuses the same SNI + HTTP/2 setup pattern, performs a full handshake, then sends a second ClientHello with the `xftp-web-hello` header on the same connection and verifies the server responds with a valid ServerHandshake (same `sessionId`), then completes the second handshake.
Add `testWebReHandshake` next to the existing `testWebHandshake` (line 504). It reuses the same SNI + HTTP/2 setup pattern, performs a full handshake, then sends a second ClientHello with the `xftp-web-hello` header on the same connection and verifies the router responds with a valid ServerHandshake (same `sessionId`), then completes the second handshake.
```haskell
-- Register in xftpServerTests (after line 86):
@@ -170,7 +170,7 @@ The only difference from `testWebHandshake`: the second `helloReq2` passes `[("x
## 4. Implementation Plan
### Step 1: Server — parameterize `processHello`
### Step 1: Router — parameterize `processHello`
Apply the diff from Section 3.1 to `src/Simplex/FileTransfer/Server.hs`.
@@ -216,6 +216,6 @@ Tab A (upload) and Tab B (download) share the same HTTP/2 connection.
## 6. Security Considerations
- **No new key material**: Re-handshake reuses existing `serverPrivKey`. No opportunity for key confusion or downgrade.
- **Identity re-verification**: Server re-signs the web challenge with its long-term signing key. Client verifies identity again.
- **Header cannot escalate privileges**: The header only triggers re-handshake (which the server was already capable of doing on first connection). It does not bypass any authentication.
- **Identity re-verification**: Router re-signs the web challenge with its long-term signing key. Client verifies identity again.
- **Header cannot escalate privileges**: The header only triggers re-handshake (which the router was already capable of doing on first connection). It does not bypass any authentication.
- **Timing**: Re-handshake takes the same code path as initial handshake, so timing side-channels are unchanged.
@@ -2,13 +2,13 @@
## 1. Problem Statement
The XFTP web client is fundamentally fragile: any transient error (browser opening a new HTTP/2 connection, network hiccup, server restart) causes an unrecoverable failure with a cryptic error message. There is no retry logic, no fetch timeout, no error categorization, and the upload uses a single server instead of distributing chunks across preset servers. This makes the app frustrating — it works most of the time but fails unpredictably, which is worse than being completely broken.
The XFTP web client is fundamentally fragile: any transient error (browser opening a new HTTP/2 connection, network hiccup, router restart) causes an unrecoverable failure with a cryptic error message. There is no retry logic, no fetch timeout, no error categorization, and the upload uses a single router instead of distributing data packets across preset routers. This makes the app frustrating — it works most of the time but fails unpredictably, which is worse than being completely broken.
### Confirmed root cause (from diagnostic logs)
When the browser opens a new HTTP/2 connection mid-operation, the new connection has a different TLS SessionId with no handshake state in the server's `TMap SessionId Handshake`. The server's `Nothing` branch in `xftpServerHandshakeV1` (Server.hs:169) unconditionally calls `processHello`, which tries to decode the command body as `XFTPClientHello`, fails, and sends a raw padded "HANDSHAKE" error string. The client cannot parse this as a proper transmission (first byte 'H' = 72 is read as batch count), producing `"expected batch count 1, got 72"`.
When the browser opens a new HTTP/2 connection mid-operation, the new connection has a different TLS SessionId with no handshake state in the router's `TMap SessionId Handshake`. The router's `Nothing` branch in `xftpServerHandshakeV1` (Server.hs:169) unconditionally calls `processHello`, which tries to decode the command body as `XFTPClientHello`, fails, and sends a raw padded "HANDSHAKE" error string. The client cannot parse this as a proper transmission (first byte 'H' = 72 is read as batch count), producing `"expected batch count 1, got 72"`.
Server log confirming the SessionId change:
Router log confirming the SessionId change:
```
DEBUG dispatch: Accepted+command sessId="ZSo1GGETgIvjbB7CWHbvGPpbMjx_b2IlC1eTI6aKfqc="
...20 successful commands...
@@ -17,32 +17,32 @@ DEBUG dispatch: Nothing sessId="mJC7Sck9xxW5UsXoPGoUWduuHghSVgf6CnD6ZC6SBhU=" we
### Why re-handshake is required (cannot be made optional)
1. **SessionId is baked into signed command data.** `encodeAuthTransmission` signs `concat(encode(sessionId), tInner)` with Ed25519. Server's `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId`. New connection = different sessionId = signature mismatch.
2. **Server generates per-session DH keys.** `processHello` creates fresh X25519 keypair stored in `HandshakeSent`. For SMP browser clients (future), `verifyCmdAuth` (Protocol.hs:1322) requires the matching `serverPrivKey` from `thAuth`.
1. **SessionId is baked into signed command data.** `encodeAuthTransmission` signs `concat(encode(sessionId), tInner)` with Ed25519. Router's `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId`. New connection = different sessionId = signature mismatch.
2. **Router generates per-session DH keys.** `processHello` creates fresh X25519 keypair stored in `HandshakeSent`. For SMP browser clients (future), `verifyCmdAuth` (Protocol.hs:1322) requires the matching `serverPrivKey` from `thAuth`.
3. **This applies to both XFTP and future SMP browser clients** — the session management approach is the same.
### Why multiple preset servers cannot work
### Why multiple preset routers cannot work
Upload (`agent.ts:105-157`) takes a single `server: XFTPServer` parameter and uploads ALL chunks to it. `web/upload.ts:133` calls `pickRandomServer(servers)` which selects ONE random server from all presets. The multi-server preset configuration is pointless — only one server is ever used per upload. The design intent (RFC section 11.6: "upload in parallel to 8 randomly selected servers") is not implemented. This must be fixed in Phase 2 (section 3.7).
Upload (`agent.ts:105-157`) takes a single `server: XFTPServer` parameter and uploads ALL data packets to it. `web/upload.ts:133` calls `pickRandomServer(servers)` which selects ONE random router from all presets. The multi-router preset configuration is pointless — only one router is ever used per upload. The design intent (RFC section 11.6: "upload in parallel to 8 randomly selected routers") is not implemented. This must be fixed in Phase 2 (section 3.7).
## 2. Solution Summary
### Phase 1: Error handling and connection resilience
1. **Server: strict dispatch for allowed protocol combinations** — reject all invalid combinations
1. **Router: strict dispatch for allowed protocol combinations** — reject all invalid combinations
2. **Client: automatic retry with re-handshake** on SESSION/HANDSHAKE errors
3. **Client: fetch timeout** with configurable duration
4. **UI: error categorization and retry** — auto-retry temporary, human-readable permanent
5. **Client: connection state with Promise-based lock and per-server queues**`ServerConnection` with `client: Promise<XFTPClient>` + `queue: Promise<void>`
5. **Client: connection state with Promise-based lock and per-router queues**`ServerConnection` with `client: Promise<XFTPClient>` + `queue: Promise<void>`
6. **Client: fix cache key** — include keyHash
### Phase 2: Multi-server upload (after Phase 1)
### Phase 2: Multi-router upload (after Phase 1)
7. **Multi-server upload with server selection and failover** — distribute chunks across servers, retry FNEW on different server if one fails
7. **Multi-router upload with router selection and failover** — distribute data packets across routers, retry FNEW on different router if one fails
## 3. Detailed Technical Design
### 3.1 Server: strict dispatch for allowed protocol combinations
### 3.1 Router: strict dispatch for allowed protocol combinations
**Principle:** Everything not explicitly done by existing Haskell/TS clients is prohibited. It is better to fail on impossible combinations than to be permissive — permissiveness complicates debugging and creates attack vectors via unexpected behaviors.
@@ -88,14 +88,14 @@ Nothing
| `FRErr SESSION` | Temporary | Yes (auto) | "Session expired, reconnecting..." |
| `FRErr HANDSHAKE` | Temporary | Yes (auto) | "Connection interrupted, reconnecting..." |
| `fetch()` TypeError | Temporary | Yes (auto) | "Network error, retrying..." |
| AbortError (timeout) | Temporary | Yes (auto) | "Server timeout, retrying..." |
| AbortError (timeout) | Temporary | Yes (auto) | "Router timeout, retrying..." |
| `FRErr AUTH` | Permanent | No | "File is invalid, expired, or has been removed" |
| `FRErr NO_FILE` | Permanent | No | "File not found — it may have expired" |
| `FRErr SIZE` | Permanent | No | "File size exceeds server limit" |
| `FRErr QUOTA` | Permanent | No | "Server storage quota exceeded" |
| `FRErr BLOCKED` | Permanent | No | "File has been blocked by server" |
| `FRErr SIZE` | Permanent | No | "File size exceeds router limit" |
| `FRErr QUOTA` | Permanent | No | "Router storage quota exceeded" |
| `FRErr BLOCKED` | Permanent | No | "File has been blocked by router" |
| `FRErr DIGEST` | Permanent | No | "File integrity check failed" |
| `FRErr INTERNAL` | Permanent | No | "Server internal error" |
| `FRErr INTERNAL` | Permanent | No | "Router internal error" |
| `CMD *` | Permanent | No | "Protocol error" |
**Retry behavior:**
@@ -156,7 +156,7 @@ if (raw.length < 20) {
2. **FRErr classification** (replaces current unconditional throw):
```typescript
// After decodeResponse, instead of throw new Error("Server error: " + err.type):
// After decodeResponse, instead of throw new Error("Router error: " + err.type):
if (response.type === "FRErr") {
const err = response.err
if (err.type === "SESSION" || err.type === "HANDSHAKE") {
@@ -206,30 +206,30 @@ Default: 30s for production, 5s for tests. Threaded through `connectXFTP` → `c
**Behavior (Option D):**
- **Temporary errors:** Auto-retry loop (3 attempts). After 3 failures, show human-readable diagnosis with manual retry button. Diagnosis examples: "Server timeout — the server may be temporarily unavailable", "Connection interrupted — your network may be unstable".
- **Temporary errors:** Auto-retry loop (3 attempts). After 3 failures, show human-readable diagnosis with manual retry button. Diagnosis examples: "Router timeout — the router may be temporarily unavailable", "Connection interrupted — your network may be unstable".
- **Permanent errors:** Show human-readable error immediately, NO retry button. User can reload page if they want to retry. Examples: "File is invalid, expired, or has been removed" (AUTH), "File not found" (NO_FILE).
**Current UI retry buttons:**
- `upload.ts:73-75` — retry calls `startUpload(pendingFile)` from scratch
- `download.ts:60` — retry calls `startDownload()` from scratch
**Improvement:** Track uploaded/downloaded chunk indices. On manual retry, skip completed chunks:
**Improvement:** Track uploaded/downloaded data packet indices. On manual retry, skip completed data packets:
```typescript
// Upload: track which chunks completed
// Upload: track which data packets completed
const completedChunks: Set<number> = new Set()
for (let i = 0; i < specs.length; i++) {
if (completedChunks.has(i)) continue
// ... create + upload chunk
// ... create + upload data packet
completedChunks.add(i)
}
// Download: already naturally resumable — each chunk is independent
// Download: already naturally resumable — each data packet is independent
```
### 3.5 Client: connection state with Promise-based lock and per-server queues
### 3.5 Client: connection state with Promise-based lock and per-router queues
**Design:** Each server gets a `ServerConnection` record containing a `Promise<XFTPClient>` (the connection lock) and a `Promise<void>` (the sequential command queue). The `XFTPClientAgent` maps server keys to these records.
**Design:** Each router gets a `ServerConnection` record containing a `Promise<XFTPClient>` (the connection lock) and a `Promise<void>` (the sequential command queue). The `XFTPClientAgent` maps router keys to these records.
The promise IS the lock — every consumer awaits the same promise. When reconnect is needed, the promise is replaced atomically.
@@ -325,7 +325,7 @@ function removeStaleConnection(
}
```
**Per-server sequential queue:** `queue` is a `Promise<void>` — the tail of the sequential operation chain. Each new operation `.then()`s onto it. It's `void` because callers hold their own typed promises; the queue only tracks completion order:
**Per-router sequential queue:** `queue` is a `Promise<void>` — the tail of the sequential operation chain. Each new operation `.then()`s onto it. It's `void` because callers hold their own typed promises; the queue only tracks completion order:
```typescript
async function enqueueCommand<T>(
@@ -348,9 +348,9 @@ async function enqueueCommand<T>(
}
```
Commands to the same server execute one at a time via the queue. Commands to different servers execute concurrently because each has its own queue. `enqueueCommand` provides sequencing; `sendXFTPCommand` (called inside `fn` via command wrappers) provides retry. They compose as: `enqueueCommand` sequences calls to wrappers that internally use `sendXFTPCommand`.
Commands to the same router execute one at a time via the queue. Commands to different routers execute concurrently because each has its own queue. `enqueueCommand` provides sequencing; `sendXFTPCommand` (called inside `fn` via command wrappers) provides retry. They compose as: `enqueueCommand` sequences calls to wrappers that internally use `sendXFTPCommand`.
**Download change:** Group chunks by server, process each server's chunks sequentially, servers in parallel. Uses `for` loop for per-server sequencing (same pattern as Stage 2 upload). `enqueueCommand` is available for cases where different callers target the same server.
**Download change:** Group data packets by router, process each router's data packets sequentially, routers in parallel. Uses `for` loop for per-router sequencing (same pattern as Stage 2 upload). `enqueueCommand` is available for cases where different callers target the same router.
```typescript
const byServer = new Map<string, FileChunk[]>()
@@ -374,7 +374,7 @@ await Promise.all([...byServer.entries()].map(async ([srv, chunks]) => {
### 3.6 Fix cache key
**Bug:** `getXFTPServerClient` (client.ts:110) uses `"https://" + server.host + ":" + server.port` as cache key, ignoring `keyHash`. Two servers with same host:port but different keyHash share a cached connection, bypassing identity verification.
**Bug:** `getXFTPServerClient` (client.ts:110) uses `"https://" + server.host + ":" + server.port` as cache key, ignoring `keyHash`. Two routers with same host:port but different keyHash share a cached connection, bypassing identity verification.
**Fix:** Use `formatXFTPServer(server)` as cache key (includes keyHash). Already available in `protocol/address.ts:52-54`.
@@ -388,11 +388,11 @@ const key = formatXFTPServer(server)
Note: With the redesign in 3.5, the cache key fix is inherent — the `connections` Map uses `formatXFTPServer(server)` everywhere.
### 3.7 Phase 2: Multi-server upload with server selection and failover
### 3.7 Phase 2: Multi-router upload with router selection and failover
**Problem:** Current upload (`agent.ts:105-157`) takes a single `server: XFTPServer` and uploads ALL chunks to it. The 12 preset servers (6 SimpleX + 6 Flux) are pointless — only one is ever used.
**Problem:** Current upload (`agent.ts:105-157`) takes a single `server: XFTPServer` and uploads ALL data packets to it. The 12 preset routers (6 SimpleX + 6 Flux) are pointless — only one is ever used.
**Design goal:** Distribute chunks across servers. Retry FNEW on a different server if one fails. Once working servers are found, prefer them (heuristic: server unlikely to fail mid-process, more likely to be broken initially due to maintenance/downtime).
**Design goal:** Distribute data packets across routers. Retry FNEW on a different router if one fails. Once working routers are found, prefer them (heuristic: router unlikely to fail mid-process, more likely to be broken initially due to maintenance/downtime).
**Reference implementation:** Haskell `Agent.hs:457-486` (`createChunk` / `createWithNextSrv`) + `Client.hs:2335-2385` (`getNextServer_` / `withNextSrv`).
@@ -400,13 +400,13 @@ Note: With the redesign in 3.5, the cache key fix is inherent — the `connectio
Two-stage architecture:
1. **Allocate stage (serial per file in Haskell):** For each chunk, call FNEW on a randomly-selected server. If FNEW fails, pick a different server and retry. Track tried hosts to avoid retrying the same server. After all chunks are assigned to servers, spawn one upload worker per server.
1. **Allocate stage (serial per file in Haskell):** For each data packet, call FNEW on a randomly-selected router. If FNEW fails, pick a different router and retry. Track tried hosts to avoid retrying the same router. After all data packets are assigned to routers, spawn one upload worker per router.
2. **Upload stage (parallel per server):** Each server worker uploads its assigned chunks sequentially (FPUT). On FPUT failure, retry on the same server with backoff (because the chunk replica already exists on that server). No server failover for FPUT.
2. **Upload stage (parallel per router):** Each router worker uploads its assigned data packets sequentially (FPUT). On FPUT failure, retry on the same router with backoff (because the data packet replica already exists on that router). No router failover for FPUT.
Server selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350):
1. Prefer servers from unused operators (operator diversity)
2. Prefer servers with unused hosts (host diversity)
Router selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350):
1. Prefer routers from unused operators (operator diversity)
2. Prefer routers with unused hosts (host diversity)
3. Random pick from the most-constrained candidate set
4. If all exhausted, reset tried set and start over
@@ -414,17 +414,17 @@ Server selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350
The web client doesn't have operators or a database. Simplified algorithm with two stages:
**Stage 1 — Allocate:** Create chunk records on servers (FNEW). Unlike Haskell which is serial here, web FNEW runs concurrently within a concurrency limit. FNEW is a small command — concurrent FNEW on the same connection is not a problem, and concurrent FNEW across servers improves upload startup time.
**Stage 1 — Allocate:** Create data packet records on routers (FNEW). Unlike Haskell which is serial here, web FNEW runs concurrently within a concurrency limit. FNEW is a small command — concurrent FNEW on the same connection is not a problem, and concurrent FNEW across routers improves upload startup time.
**Stage 2 — Upload:** Upload chunk data (FPUT). Parallel across servers, sequential per server (reuses per-server queues from 3.5). FPUT retries on the same server with backoff — no server rotation because the chunk replica already exists on that server. Stage 2 reads chunk data by offset (via `readChunk`), so `SentChunk` must be extended with `chunkOffset: number` (from ChunkSpec).
**Stage 2 — Upload:** Upload data packet content (FPUT). Parallel across routers, sequential per router (reuses per-router queues from 3.5). FPUT retries on the same router with backoff — no router rotation because the data packet replica already exists on that router. Stage 2 reads data packet content by offset (via `readChunk`), so `SentChunk` must be extended with `chunkOffset: number` (from ChunkSpec).
```typescript
interface UploadState {
untriedServers: XFTPServer[] // servers not yet attempted — initially all servers
workingServers: XFTPServer[] // servers that succeeded FNEW
untriedServers: XFTPServer[] // routers not yet attempted — initially all routers
workingServers: XFTPServer[] // routers that succeeded FNEW
}
const MAX_FNEW_ATTEMPTS = 5 // per chunk: try up to 5 different servers
const MAX_FNEW_ATTEMPTS = 5 // per data packet: try up to 5 different routers
async function uploadFile(
agent: XFTPClientAgent,
@@ -455,7 +455,7 @@ async function uploadFile(
)
await Promise.all(allocateWorkers)
// Stage 2: Upload — parallel across servers, sequential per server
// Stage 2: Upload — parallel across routers, sequential per router
// readChunk reads from the encrypted file by offset (same as Phase 1 uploadFile)
let uploaded = 0
const total = encrypted.chunkSizes.reduce((a, b) => a + b, 0)
@@ -473,7 +473,7 @@ async function uploadFile(
}
```
**`createChunkWithFailover`** — server selection with per-chunk retry limit:
**`createChunkWithFailover`** — router selection with per-data-packet retry limit:
```typescript
async function createChunkWithFailover(
@@ -515,7 +515,7 @@ function pickServer(
state: UploadState,
concurrency: number
): XFTPServer {
// Once enough working servers found, only use those
// Once enough working routers found, only use those
if (state.workingServers.length >= concurrency) {
return randomPick(state.workingServers)
}
@@ -524,7 +524,7 @@ function pickServer(
const idx = Math.floor(Math.random() * state.untriedServers.length)
return state.untriedServers.splice(idx, 1)[0] // remove from untried
}
// All tried — reset untried to non-working servers and retry
// All tried — reset untried to non-working routers and retry
state.untriedServers = allServers.filter(
s => !state.workingServers.some(w => formatXFTPServer(w) === formatXFTPServer(s))
)
@@ -532,22 +532,22 @@ function pickServer(
const idx = Math.floor(Math.random() * state.untriedServers.length)
return state.untriedServers.splice(idx, 1)[0]
}
// Every server is working — pick any working
// Every router is working — pick any working
return randomPick(state.workingServers)
}
```
**Algorithm:** Two lists — `untriedServers` (initially all) and `workingServers` (initially empty). When `workingServers.length < concurrency`, pick from `untriedServers` (removing on pick). On FNEW success, add to `workingServers`. On FNEW failure, server is already removed from `untriedServers`; remove from `workingServers` if present. When `untriedServers` is empty, reset it to all non-working servers. Once `workingServers.length >= concurrency`, pick randomly only from `workingServers`.
**Algorithm:** Two lists — `untriedServers` (initially all) and `workingServers` (initially empty). When `workingServers.length < concurrency`, pick from `untriedServers` (removing on pick). On FNEW success, add to `workingServers`. On FNEW failure, router is already removed from `untriedServers`; remove from `workingServers` if present. When `untriedServers` is empty, reset it to all non-working routers. Once `workingServers.length >= concurrency`, pick randomly only from `workingServers`.
**Termination condition:** Each chunk tries at most `min(serverCount, 5)` different servers. If all attempts fail, the chunk fails and the upload fails with the last error. Rationale: if 5 out of 12 servers are down, something systemic is wrong and continuing is unlikely to help. Timeouts count as failures — the timed-out server is removed from working and a different server is picked next.
**Termination condition:** Each data packet tries at most `min(routerCount, 5)` different routers. If all attempts fail, the data packet fails and the upload fails with the last error. Rationale: if 5 out of 12 routers are down, something systemic is wrong and continuing is unlikely to help. Timeouts count as failures — the timed-out router is removed from working and a different router is picked next.
**Key differences from Haskell:**
- No operator concept — just host diversity via random selection
- No database — state tracked in-memory during upload
- FNEW runs concurrently (Haskell is serial) — improves startup time
- FNEW is cheap and retried with server rotation; FPUT retries on same server
- FNEW is cheap and retried with router rotation; FPUT retries on same router
**Download changes (also Phase 2):** Default concurrency should be 4 (matching Haskell). Download already groups by server in 3.5. If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc. (fallback across replicas).
**Download changes (also Phase 2):** Default concurrency should be 4 (matching Haskell). Download already groups by router in 3.5. If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc. (fallback across replicas).
## 4. Implementation Plan
@@ -560,7 +560,7 @@ Steps are ordered by dependency and should be implemented one by one.
- Add import for `formatXFTPServer`
- Run existing tests to verify no regression
#### Step 2: Typed error detection for padded server errors (3.2 client-side)
#### Step 2: Typed error detection for padded router errors (3.2 client-side)
- Add `XFTPRetriableError` class
- In `sendXFTPCommand`, detect padded error strings before `decodeTransmission`
- Classify `FRErr` responses as retriable or permanent with human-readable messages
@@ -573,16 +573,16 @@ Steps are ordered by dependency and should be implemented one by one.
- Add vitest test: timeout triggers after configured duration
- Run existing tests
#### Step 4: Connection state with Promise-based lock and per-server queues (3.5)
#### Step 4: Connection state with Promise-based lock and per-router queues (3.5)
- Introduce `ServerConnection` record: `{client: Promise<XFTPClient>, queue: Promise<void>}`
- Replace `XFTPClientAgent.clients: Map<string, XFTPClient>` with `connections: Map<string, ServerConnection>`
- Implement `reconnectClient` — replaces `conn.client` with new promise, preserves queue
- Implement `enqueueCommand` — chains operation onto server's queue
- Implement `enqueueCommand` — chains operation onto router's queue
- Implement `removeStaleConnection` — removes entry only if current promise is the failed one
- Auto-cleanup: `p.catch(() => delete)` removes failed connections so next caller starts fresh
- Adapt `closeXFTPServerClient` and `closeXFTPAgent`
- Add vitest tests:
- Concurrent calls to same server produce single connection
- Concurrent calls to same router produce single connection
- Failed promise is cleaned up, next caller gets fresh connection
#### Step 5: Automatic retry in sendXFTPCommand (3.2)
@@ -594,61 +594,61 @@ Steps are ordered by dependency and should be implemented one by one.
- Max 3 retries for retriable errors, immediate throw for permanent
- On retriable error: call `reconnectClient` and retry. On retriable error exhausted: call `removeStaleConnection` to clean up. On permanent error: throw immediately without touching connection
- Add vitest tests:
- Server started with delay → first attempt fails, retry succeeds
- Router started with delay → first attempt fails, retry succeeds
- 3 retries exhausted → error propagates with human-readable message
- Non-retriable error (AUTH) → no retry, immediate failure
#### Step 6: Server-side stale session handling (3.1)
#### Step 6: Router-side stale session handling (3.1)
- Add one guard to `Nothing` branch: `sniUsed && not webHello -> throwE SESSION`
- Remove debug `hPutStrLn stderr` lines (all 6 occurrences in dispatch)
- All other branches unchanged
- Run Haskell tests + Playwright tests
#### Step 7: Download with per-server grouping
- Modify `downloadFileRaw` to group chunks by server, sequential within each server (`for` loop), parallel across servers (`Promise.all`)
- Add vitest test: concurrent downloads from different servers run in parallel
#### Step 7: Download with per-router grouping
- Modify `downloadFileRaw` to group data packets by router, sequential within each router (`for` loop), parallel across routers (`Promise.all`)
- Add vitest test: concurrent downloads from different routers run in parallel
#### Step 8: UI error improvements (3.4)
- Temporary errors: auto-retry loop (3 attempts), then show human-readable diagnosis + manual retry button
- Permanent errors: show human-readable error, NO retry button
- Manual retry resumes from last successful chunk (not full restart)
- Manual retry resumes from last successful data packet (not full restart)
#### Step 9: Remove debug logging
- Remove all `console.log('[DEBUG ...]')` and `hPutStrLn stderr "DEBUG ..."` lines
- Keep `console.error('[XFTP] ...')` error logging
### Phase 2: Multi-server upload
### Phase 2: Multi-router upload
Implement after Phase 1 is complete and tested.
#### Step 10: Multi-server upload with failover (3.7)
- Extend `SentChunk` with `chunkOffset: number` (from ChunkSpec) and `server: XFTPServer` (assigned during allocate) — Stage 2 reads data by offset and groups chunks by server
#### Step 10: Multi-router upload with failover (3.7)
- Extend `SentChunk` with `chunkOffset: number` (from ChunkSpec) and `server: XFTPServer` (assigned during allocate) — Stage 2 reads data by offset and groups data packets by router
- Change `uploadFile` signature: takes `allServers: XFTPServer[]` instead of single `server`
- Implement `UploadState` with `untriedServers` and `workingServers`
- Implement `createChunkWithFailover` and `pickServer`: two-list selection (untried → working once enough found), max `min(serverCount, 5)` attempts per chunk
- Implement `createChunkWithFailover` and `pickServer`: two-list selection (untried → working once enough found), max `min(routerCount, 5)` attempts per data packet
- Allocate stage: concurrent FNEW within concurrency limit (default 4)
- Upload stage: parallel across servers, sequential per server (reuse queue from Step 7)
- Upload stage: parallel across routers, sequential per router (reuse queue from Step 7)
- Update `web/upload.ts`: pass `getServers()` instead of `pickRandomServer(getServers())`
- Update description building: each chunk references its actual server
- Update description building: each data packet references its actual router
- Add vitest tests:
- File split across N servers (verify different servers in description)
- One server down → chunks redistributed to others
- All servers down → error after exhausting 5 attempts per chunk
- File split across N routers (verify different routers in description)
- One router down → data packets redistributed to others
- All routers down → error after exhausting 5 attempts per data packet
#### Step 11: Download concurrency and replica fallback
- Change default download concurrency from 1 to 4
- If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc.
- Uses per-server queues from Step 7
- Uses per-router queues from Step 7
## 5. Testing Plan
### Principle
Prefer low-level vitest tests over Playwright E2E. Each new function gets one focused test. Pure functions tested without mocks; connection management tested with mock `connectXFTP`; server behavior tested with real server. Total: 13 tests across 4 files.
Prefer low-level vitest tests over Playwright E2E. Each new function gets one focused test. Pure functions tested without mocks; connection management tested with mock `connectXFTP`; router behavior tested with real router. Total: 13 tests across 4 files.
Tests A-C run in browser context (`@vitest/browser` with Chromium headless), configured in `vitest.config.ts`. Test D (integration) requires a separate Node.js vitest config since it uses `node:http2`. Existing `globalSetup.ts` provides a real XFTP server for integration tests.
Tests A-C run in browser context (`@vitest/browser` with Chromium headless), configured in `vitest.config.ts`. Test D (integration) requires a separate Node.js vitest config since it uses `node:http2`. Existing `globalSetup.ts` provides a real XFTP router for integration tests.
### Test file A: `test/errors.test.ts` — pure, no server
### Test file A: `test/errors.test.ts` — pure, no router
Tests error classification and padded error detection (Steps 2, 5).
@@ -682,7 +682,7 @@ expect(re.message).toContain("expired") // "Session expired, reconnecting..."
**T3. Padded error detection extracts error string from padded block**
```typescript
import {blockPad, blockUnpad} from '../src/protocol/transmission.js'
// Simulate server sending padded "SESSION"
// Simulate router sending padded "SESSION"
const padded = blockPad(new TextEncoder().encode("SESSION"))
const raw = blockUnpad(padded)
expect(raw.length).toBeLessThan(20)
@@ -694,7 +694,7 @@ const normalRaw = blockUnpad(normalBlock)
expect(normalRaw.length).toBeGreaterThan(20) // not mistaken for padded error
```
### Test file B: `test/connection.test.ts` — mock connectXFTP, no server
### Test file B: `test/connection.test.ts` — mock connectXFTP, no router
Tests connection management functions (Steps 4, 5). Uses `vi.mock` to replace `connectXFTP` with a controllable promise factory.
@@ -800,7 +800,7 @@ await expect(sendXFTPCommand(agent3, server, dummyKey, dummyId, encodePING()))
expect(vi.mocked(connectXFTP)).toHaveBeenCalledTimes(1) // initial only, no reconnect
```
### Test file C: `test/server-selection.test.ts` — pure, no server
### Test file C: `test/server-selection.test.ts` — pure, no router
Tests `pickServer` state machine (Step 10). Determinism: seed `Math.random` or test invariants not specific picks.
@@ -833,12 +833,12 @@ const state: UploadState = {
workingServers: [s1, s2] // only 2 working, concurrency=4
}
const picked = pickServer(servers, state, 4)
// Should have reset untried to non-working servers and picked from them
// Should have reset untried to non-working routers and picked from them
expect([s3, s4, s5]).toContainEqual(picked)
expect(state.untriedServers.length).toBe(2) // 3 non-working minus 1 picked
```
### Test file D: `test/integration.test.ts` — real server, Node.js mode
### Test file D: `test/integration.test.ts` — real router, Node.js mode
Requires separate vitest config with `browser: {enabled: false}` since these tests use `node:http2` directly. Alternatively, add `test/vitest.node.config.ts` that includes only `test/integration.test.ts` and runs in Node.js.
@@ -847,10 +847,10 @@ Requires separate vitest config with `browser: {enabled: false}` since these tes
import http2 from 'node:http2'
// Connect and handshake normally via the client
const client = await connectXFTP(server)
// Create a raw HTTP/2 session (new TLS SessionId, no handshake state on server)
// Create a raw HTTP/2 session (new TLS SessionId, no handshake state on router)
const session = http2.connect(client.baseUrl, {rejectUnauthorized: false})
// Build a dummy command block using the old client's sessionId.
// Content doesn't matter — server detects stale session before parsing command.
// Content doesn't matter — router detects stale session before parsing command.
const dummyKey = new Uint8Array(64) // Ed25519 private key (dummy)
const dummyId = new Uint8Array(24) // entity ID (dummy)
const cmdBlock = encodeAuthTransmission(client.sessionId, new Uint8Array(0), dummyId, encodePING(), dummyKey)
@@ -862,7 +862,7 @@ const resp = await new Promise<Uint8Array>((resolve, reject) => {
req.on("error", reject)
req.end(Buffer.from(cmdBlock))
})
// Server should return padded "SESSION" (not crash, not "HANDSHAKE")
// Router should return padded "SESSION" (not crash, not "HANDSHAKE")
const raw = blockUnpad(resp.subarray(0, XFTP_BLOCK_SIZE))
expect(new TextDecoder().decode(raw)).toBe("SESSION")
session.close()
@@ -885,7 +885,7 @@ await expect(
| Cache key fix (Step 1) | Existing round-trip test — uses `formatXFTPServer` after refactor |
| Basic upload/download | 24 Playwright tests + 1 vitest browser test |
| File size limits, unicode filenames | Playwright edge case tests |
| Server startup/teardown | `globalSetup.ts` / `globalTeardown.ts` |
| Router startup/teardown | `globalSetup.ts` / `globalTeardown.ts` |
| Handshake + identity verification | `connectXFTP` in existing round-trip test |
### Test ordering
@@ -895,7 +895,7 @@ Tests must be added alongside their implementation step:
- **Step 3**: Add T13 (test/integration.test.ts) — requires Node.js vitest config
- **Step 4**: Add T4, T5, T6, T7 (test/connection.test.ts)
- **Step 5**: Add T8 (test/connection.test.ts)
- **Step 6**: Add T12 (test/integration.test.ts) — requires server change + Node.js vitest config
- **Step 6**: Add T12 (test/integration.test.ts) — requires router change + Node.js vitest config
- **Step 10**: Add T9, T10, T11 (test/server-selection.test.ts)
## 6. Context for Implementation Sessions
@@ -914,30 +914,30 @@ Tests must be added alongside their implementation step:
- `web/servers.ts``getServers`, `pickRandomServer`
**TypeScript (xftp-web/test/):**
- `browser.test.ts` — vitest Node.js test template (uses real Haskell server)
- `globalSetup.ts`server startup, config generation, port file
- `browser.test.ts` — vitest Node.js test template (uses real Haskell router)
- `globalSetup.ts`router startup, config generation, port file
- `page.spec.ts` — Playwright page tests
**Haskell (reference for multi-server):**
- `src/Simplex/FileTransfer/Agent.hs``createChunk` (lines 457-486, allocate stage), `runXFTPSndPrepareWorker` (lines 391-430, serial allocate in Haskell), `runXFTPSndWorker` (lines 494-548, per-server upload worker)
**Haskell (reference for multi-router):**
- `src/Simplex/FileTransfer/Agent.hs``createChunk` (lines 457-486, allocate stage), `runXFTPSndPrepareWorker` (lines 391-430, serial allocate in Haskell), `runXFTPSndWorker` (lines 494-548, per-router upload worker)
- `src/Simplex/Messaging/Agent/Client.hs``getNextServer_` (lines 2335-2350), `withNextSrv` (lines 2366-2385), `pickServer` (lines 2309-2314)
**Haskell (server):**
**Haskell (router):**
- `src/Simplex/FileTransfer/Server.hs``xftpServerHandshakeV1` (lines 165-244), `processRequest` (lines 403-435)
- `src/Simplex/Messaging/Protocol.hs``tDecodeServer` (lines 2239-2265) — sessionId verification at line 2242
### Key design constraints
1. `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId` — commands signed with old sessionId WILL fail on new connection
2. Server generates per-session DH key in `processHello` (Server.hs:207) — cannot be shared across sessions
2. Router generates per-session DH key in `processHello` (Server.hs:207) — cannot be shared across sessions
3. `fetch()` provides zero control over HTTP/2 connection reuse — browser decides
4. `xftp-web-hello` header is only checked in dispatch (Server.hs:192), NOT inside `processHello`
5. Handshake-phase errors are raw padded strings; command-phase errors are proper ERR transmissions
6. Ed25519 signature verification (`TASignature` path, Protocol.hs:1314) does NOT use `thAuth` — but SMP will
7. Reconnect must re-handshake to get new sessionId AND new server DH key
7. Reconnect must re-handshake to get new sessionId AND new router DH key
8. The new `throwE SESSION` guard (Step 6) sends a raw padded "SESSION" string — no sessionId framing. Client detects this via padded error heuristic (section 3.2), not via sessionId mismatch
9. FNEW is cheap (creates chunk record on server) — retry with different server on failure
10. FPUT retries on same server (chunk replica already exists there) — close connection + backoff
9. FNEW is cheap (creates data packet record on router) — retry with different router on failure
10. FPUT retries on same router (data packet replica already exists there) — close connection + backoff
## 7. Plan Maintenance
@@ -12,8 +12,8 @@ Make CLI produce and consume web-compatible links so that:
- CLI `recv` accepts a web link URL as input (alternative to `.xftp` file path)
- Browser can download files uploaded by CLI and vice versa
The web page host is derived from the XFTP server address - the server that hosts the file
also hosts the download page. Making XFTP servers actually serve the web page is a separate
The web page host is derived from the XFTP router address - the router that hosts the file
also hosts the download page. Making XFTP routers actually serve the web page is a separate
concern (not covered here), but the link format anticipates it.
The YAML file description format is already identical between CLI and web.
@@ -33,7 +33,7 @@ Encoding chain (agent.ts:64-68):
3. `pako.deflateRaw(bytes)` -> compressed
4. `base64urlEncode(compressed)` -> URI fragment (no `#`)
For multi-chunk files exceeding ~400 chars in URI, a redirect description is uploaded:
For multi-packet files exceeding ~400 chars in URI, a redirect description is uploaded:
the real file description is encrypted, uploaded as a separate XFTP file, and a smaller
"redirect" description (pointing to it) is put in the URI.
@@ -111,7 +111,7 @@ Extracts the actual filename from the path and embeds it in the encrypted header
#### CLI download: uses filename from header (ok)
`Crypto.hs:62-66` (single chunk) / `Crypto.hs:72-74` (multi-chunk):
`Crypto.hs:62-66` (single data packet) / `Crypto.hs:72-74` (multi-packet):
```haskell
(FileHeader {fileName}, rest) <- parseFileHeader decryptedContent
destFile <- withExceptT FTCEFileIOError $ getDestFile fileName
@@ -163,19 +163,19 @@ The CLI should consider adding filename sanitization similar to the web client f
### 2. Web Link Host Derivation
The web page URL domain comes from the XFTP server address, not from a CLI flag:
The web page URL domain comes from the XFTP router address, not from a CLI flag:
- **Non-redirected description**: use the server host of the first chunk's first replica.
- **Non-redirected description**: use the router host of the first data packet's first replica.
E.g., `xftp://abc=@xftp1.simplex.im` -> `https://xftp1.simplex.im/#<encoded>`
- **Redirected description**: use the server host of the redirect chunk (the outer description's
chunk that stores the encrypted inner description).
- **Redirected description**: use the router host of the redirect data packet (the outer description's
data packet that stores the encrypted inner description).
The server address format is `xftp://<keyhash>@<host>[,<host2>,...][:<port>]`.
The router address format is `xftp://<keyhash>@<host>[,<host2>,...][:<port>]`.
The web link uses `https://<host>` (port 443 implied).
This means the CLI does not need a `--web-url` flag - the server address fully determines
the link. The XFTP server serving the web page is a separate deployment concern.
This means the CLI does not need a `--web-url` flag - the router address fully determines
the link. The XFTP router serving the web page is a separate deployment concern.
### 3. Web URI Encoding/Decoding in Haskell
@@ -196,7 +196,7 @@ decodeWebURI :: ByteString -> Either String (ValidFileDescription 'FRecipient)
-- 4. validateFileDescription
-- Build full web link from file description
-- Extracts server host from first chunk replica (or redirect chunk)
-- Extracts router host from first data packet replica (or redirect data packet)
fileWebLink :: FileDescription 'FRecipient -> (String, ByteString)
-- Returns (webHost, uriFragment)
-- Caller assembles: "https://" <> webHost <> "/#" <> uriFragment
@@ -210,20 +210,20 @@ The `zlib` Haskell package provides `Codec.Compression.Zlib.Raw` for raw DEFLATE
### 4. Redirect Description Support
The CLI currently does NOT create redirect descriptions. For single-server single-recipient
uploads, most file descriptions fit in a reasonable URI even for multi-chunk files. But for
large files (many chunks x long server hostnames), the URI can exceed practical limits.
The CLI currently does NOT create redirect descriptions. For single-router single-recipient
uploads, most file descriptions fit in a reasonable URI even for multi-packet files. But for
large files (many data packets x long router hostnames), the URI can exceed practical limits.
**Approach**: Match the web client threshold.
- After encoding the URI, if `length > 400` and chunks > 1, upload a redirect description.
- After encoding the URI, if `length > 400` and data packets > 1, upload a redirect description.
- The redirect upload uses the same XFTP upload flow: encrypt YAML -> upload as file -> create
outer description pointing to it.
- This matches `agent.ts:152-155` exactly.
- The redirect chunk's server becomes the web link host.
- The redirect data packet's router becomes the web link host.
For CLI download from a redirect URI, the existing `cliReceiveFile` needs extension:
- After decoding the file description, check `redirect` field.
- If present: download and decrypt the redirect chunks first to get the inner description,
- If present: download and decrypt the redirect data packets first to get the inner description,
then download the actual file using the inner description.
- The web client already does this (`resolveRedirect` in agent.ts:320-346).
@@ -281,16 +281,16 @@ Already identical. The web `description.ts` explicitly matches Haskell `Data.Yam
Adding a cross-client test (CLI upload -> web download, or web upload -> CLI download) would
validate interop end-to-end.
### 7. Server Compatibility
### 7. Router Compatibility
No server changes needed. Both clients use the same XFTP protocol (FGET, FPUT, FNEW, FACK, FDEL).
No router changes needed. Both clients use the same XFTP protocol (FGET, FPUT, FNEW, FACK, FDEL).
The web client adds `xftp-web-hello: 1` header for the hello handshake, but the actual file
operations are identical wire-format.
The only consideration: CLI uses native HTTP/2 (via `http2` Haskell package), web uses
browser `fetch()` API over HTTP/2. Both produce identical XFTP protocol frames.
**Note**: Making XFTP servers actually serve the web download page at `https://<host>/` is a
**Note**: Making XFTP routers actually serve the web download page at `https://<host>/` is a
separate deployment/infrastructure task. This plan only establishes the link format convention
so that links are ready to work once servers serve the page.
@@ -301,7 +301,7 @@ so that links are ready to work once servers serve the page.
1. Add `zlib` dependency to `simplexmq.cabal`
2. Add `encodeWebURI` / `decodeWebURI` / `fileWebLink` to `Simplex.FileTransfer.Description`
(or a new `Simplex.FileTransfer.Description.WebURI` module)
3. `fileWebLink` extracts host from first chunk's first replica server address
3. `fileWebLink` extracts host from first data packet's first replica router address
4. Add unit tests: encode a known FileDescription, verify output matches web client encoding
5. Add round-trip test: encode -> decode -> compare
@@ -309,7 +309,7 @@ so that links are ready to work once servers serve the page.
1. Modify `ReceiveOptions` to accept `Either FilePath WebURL` for `fileDescription`
2. In `cliReceiveFile`: if URL, extract fragment after `#`, call `decodeWebURI`
3. Add redirect resolution: if `redirect /= Nothing`, download redirect chunks,
3. Add redirect resolution: if `redirect /= Nothing`, download redirect data packets,
decrypt, parse inner description, then proceed with download
4. Test: upload via web page -> copy link -> `xftp recv <link>`
@@ -1,3 +1,10 @@
---
Proposed: 2022-07-22
Implemented: ~2022-08
Standardized: 2026-03-09
Protocol: simplex-messaging
---
# Accessing SMP servers via Tor
## Problem
@@ -1,3 +1,12 @@
---
Proposed: 2021-01-26
Implemented: ~2022
Standardized: 2026-03-09
Protocol: simplex-messaging v1, evolved through v7
---
> **Implementation note:** All cryptographic primitives changed from this proposal. Transport: TLS 1.2/1.3 replaced the custom RSA handshake. E2E: Double ratchet with AES-GCM replaced per-message RSA-OAEP encryption. Auth: Ed25519/X25519 DH-based authenticated encryption (SMP v7) replaced RSA-PSS signatures. The transmission format (signature CRLF signed) was implemented as proposed.
# SMP agent: cryptography
3 main directions of work to enable basic level of security for communication via SMP agents and servers at the current stage of the project:
@@ -1,3 +1,10 @@
---
Proposed: 2022-06-13
Implemented: ~2022-06
Standardized: 2026-03-09
Protocol: agent-protocol
---
# DB access and processing messages for iOS notification service extension
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2022-06-13
Implemented: ~2022-06
Standardized: 2026-03-09
Protocol: agent-protocol
---
sequenceDiagram
participant M as iOS message<br>notification
participant S as iOS system
@@ -1,3 +1,10 @@
---
Proposed: 2023-05-03
Implemented: 2023-07-13
Standardized: 2026-03-09
Protocol: agent-protocol v4
---
# Delivery receipts
## Problems
@@ -1,3 +1,10 @@
---
Proposed: 2024-02-03
Implemented: 2024-04-30
Standardized: 2026-03-09
Protocol: simplex-messaging v7
---
# Repudiation for message senders
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2024-06-14
Implemented: 2024-06-30
Standardized: 2026-03-09
Protocol: simplex-messaging v9, agent-protocol v6
---
# Faster connection establishment
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2024-01-26
Implemented: ~2024-01
Standardized: 2026-03-09
Protocol: xftp
---
# Sending large file descriptions
It is desirable to provide a QR code/URI from which a file can be downloaded. This way files may be addressed outside a chat client.
@@ -1,3 +1,12 @@
---
Proposed: 2021-01-20
Implemented: ~2021
Standardized: 2026-03-09
Protocol: agent-protocol
---
> **Implementation note:** Logging infrastructure exists but the format evolved from the proposed ASCII art format to structured server statistics, TLS error logging, and Prometheus metrics.
# SMP agent logging
## Problem and proposed solution.
@@ -1,3 +1,12 @@
---
Proposed: 2021-01-26
Implemented: ~2022
Standardized: 2026-03-09
Protocol: agent-protocol, simplex-messaging v2
---
> **Implementation note:** Phase 1 (agent auto-ACK, store in DB, forward to client on SUB) is implemented. The GET command was added in SMP v2 for iOS NSE message retrieval. Phases 2 and 3 (fine-grained MGET/MDEL/MACK commands and autonomous agent with background polling) were not implemented.
# SMP Agent: message management
The proposal is to change the way SMP agent manages the messages from the SMP servers.
@@ -1,3 +1,10 @@
---
Proposed: 2022-03-22
Implemented: ~2022
Standardized: 2026-03-09
Protocol: push-notifications v1
---
# Notification server
## Background and motivation
@@ -1,3 +1,10 @@
---
Proposed: 2021-05-17
Implemented: ~2021
Standardized: 2026-03-09
Protocol: agent-protocol v1
---
# Open connections
## Problem
@@ -1,3 +1,12 @@
---
Proposed: 2024-03-03
Implemented: 2024-03-14
Standardized: 2026-03-09
Protocol: agent-protocol v5
---
> **Implementation note:** PQ version negotiation and per-connection PQ mode are implemented. The proposed `RatchetVR` and `EncodingV` type class names were not adopted; the functionality was integrated through existing version range types, PQ-dependent size constants (`e2eEncConnInfoLength`, `e2eEncAgentMsgLength`), and the `pqdrSMPAgentVersion` constant.
# Migrating existing connections to post-quantum double ratchet algorithm
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2023-12-29
Implemented: 2024-03-14
Standardized: 2026-03-09
Protocol: pqdr v1, agent-protocol v5
---
# Post-quantum double ratchet implementation
See [the previous doc](https://github.com/simplex-chat/simplex-chat/blob/stable/docs/rfcs/2023-09-30-pq-double-ratchet.md).
@@ -1,3 +1,10 @@
---
Proposed: 2022-12-27
Implemented: ~2023
Standardized: 2026-03-09
Protocol: simplex-messaging, agent-protocol
---
# SMP and SMP agent protocol extensions to manage queue quotas
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2022-08-14
Implemented: ~2022
Standardized: 2026-03-09
Protocol: agent-protocol v2
---
# SMP queue rotation and redundancy
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2023-10-25
Implemented: ~2024
Standardized: 2026-03-09
Protocol: xrcp v1
---
# SimpleX Remote Control protocol
Using profiles in SimpleX Chat mobile app from desktop app with minimal risk to the security/threat model of SimpleX protocols.
@@ -1,3 +1,12 @@
---
Proposed: 2023-05-02
Implemented: 2023-06-30
Standardized: 2026-03-09
Protocol: agent-protocol v3
---
> **Implementation note:** Early brainstorm document. The implementation followed the more detailed RFC 2023-06-08-resync-ratchets, which refined the state machine to use a single RatchetSyncState (RSOk/RSAllowed/RSRequired/RSStarted/RSAgreed) and defined the AgentRatchetKey envelope type.
# Re-sync encryption ratchets, queue rotation, message delivery receipts
This is very unfocussed doc outlining several problems that seem somewhat related, and some possible solution approaches.
@@ -1,3 +1,10 @@
---
Proposed: 2023-06-08
Implemented: 2023-06-30
Standardized: 2026-03-09
Protocol: agent-protocol v3
---
# Re-sync encryption ratchets
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2023-09-12
Implemented: 2024-06-21
Standardized: 2026-03-09
Protocol: simplex-messaging v8
---
# Protecting IP addresses of the users from their contacts
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2022-12-26
Implemented: ~2023
Standardized: 2026-03-09
Protocol: xftp v1
---
# SimpleX File Transfer protocol
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2022-11-11
Implemented: 2022-11-12
Standardized: 2026-03-09
Protocol: simplex-messaging v5
---
# SMP Basic Auth
## Problem
@@ -1,3 +1,12 @@
---
Proposed: 2023-05-24
Implemented: 2024-06-21
Standardized: 2026-03-09
Protocol: simplex-messaging v8
---
> **Implementation note:** Short conceptual proposal. The full design evolved into the two-hop onion routing architecture described in RFC 2023-09-12-second-relays, implemented as SMP v8 with PRXY/PKEY/PFWD/RFWD/RRES/PRES commands.
# SMP and XFTP delivery relays
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2022-06-05
Implemented: 2022-06-06
Standardized: 2026-03-09
Protocol: simplex-messaging v2
---
# SMP protocol changes to support push notifications on iOS
## Problem
@@ -1,3 +1,10 @@
---
Proposed: 2024-03-28
Implemented: ~2024
Standardized: 2026-03-09
Protocol: xftp v2
---
# XFTP version agreement
## Problem
@@ -1,3 +1,12 @@
---
Proposed: 2024-06-01
Implemented: ~2024
Standardized: 2026-03-10
Protocol: agent-protocol
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Evolving agent API
## Problem
@@ -1,12 +1,21 @@
# Service certificates for high volume servers and services connecting to SMP servers
---
Proposed: 2025-05-05
Implemented: ~2025 (SMP v16)
Standardized: 2026-03-10
Protocol: simplex-messaging
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Service certificates for high volume routers and services connecting to SMP routers
## Problem
The absense of user and client identification benefits privacy, but it requires separately authorizing subscription for each messaging queue, that doesn't scale when a high volume server or service acts as a client for SMP server even for the current traffic and network size.
The absence of user and client identification benefits privacy, but it requires separately authorizing subscription for each messaging queue, that doesn't scale when a high volume router or service acts as a client for SMP router even for the current traffic and network size.
These servers/services include:
These routers/services include:
- operators' chat relays (aka super-peers),
- notification servers,
- notification routers,
- high-traffic service chat bots,
- high-traffic business support clients.
@@ -16,31 +25,31 @@ Self-hosted chat relays may want to retain privacy, so they will not use client
Even today, directory service subscribing to all queues may take 15-20 minutes, which is experienced as downtime by the end users.
Notification servers also acting as clients to messaging servers also take 15-20 minutes to subscribe to all notifications, during which time notifications are not delivered.
Notification routers also acting as clients to messaging routers also take 15-20 minutes to subscribe to all notifications, during which time notifications are not delivered.
Not only these subscription take a lot of time, they also consume a large amount of memory both in the clients and in the servers, as association between clients and queues is currently session-scoped and not persisted anywhere (and it should not be, because end-users' clients do need privacy).
Not only these subscriptions take a lot of time, they also consume a large amount of memory both in the clients and in the routers, as association between clients and queues is currently session-scoped and not persisted anywhere (and it should not be, because end-users' clients do need privacy).
## Solution
High volume "clients" (operators' chat relays, directory service, SimpleX Chat team support client, SimpleX Status bot, etc.) that don't need privacy will identify themselves to the messaging servers at a point of connection by providing client sertificate, both in TLS handshake and in SMP handshake (the same certificate must be provided).
High volume "clients" (operators' chat relays, directory service, SimpleX Chat team support client, SimpleX Status bot, etc.) that don't need privacy will identify themselves to the messaging routers at a point of connection by providing client certificate, both in TLS handshake and in SMP handshake (the same certificate must be provided).
All the new queues and subscriptions made in this session will be creating a permanent association of the messaging queue with the client, and on subsequent reconnections the client can "subscribe" to all their queues with a single client subscription command.
This will save a lot of time subscribing and resubscribing on server and client restarts, servers' bandwidth, servers' traffic spikes, and memory of both clients and servers.
This will save a lot of time subscribing and resubscribing on router and client restarts, routers' bandwidth, routers' traffic spikes, and memory of both clients and routers.
## Protocol
An ephemeral per-session signature key signed by long-term client certificate is used for client authorization this session signature key will be passed in SMP handshake.
An ephemeral per-session signature key signed by long-term client certificate is used for client authorization -- this session signature key will be passed in SMP handshake.
To transition existing queues, the subscription command will have to be double-signed - by the queue key, and then by client key.
When server receives such "hand-over" subscription it would create a permanent association between the client certificate and the queue, and on subsequent re-connections the client can subscribe to all the existing queues still associated with the client with one command.
When router receives such "hand-over" subscription it would create a permanent association between the client certificate and the queue, and on subsequent re-connections the client can subscribe to all the existing queues still associated with the client with one command.
The server will respond to the client with the number of queues it was subscribed to - it would both inform the client that it has to re-connect in case of interruption, and can be used for client and server statistics.
The router will respond to the client with the number of queues it was subscribed to - it would both inform the client that it has to re-connect in case of interruption, and can be used for client and router statistics.
When client creates a new queue, it would also sign the request with both keys, per-queue and client's. Other queue operations (e.g., deletion, or changing associated queue data for short links) would still require two signatures, both the queue key and the client key.
The open question is whether there is any value in allowing to remove the association between the client and the queue. Probably not, as threat model should assume that the server would retain this information, and the use-case for users controlling their servers is narrow.
The open question is whether there is any value in allowing to remove the association between the client and the queue. Probably not, as threat model should assume that the router would retain this information, and the use-case for users controlling their routers is narrow.
## Protocol connection handshake
@@ -69,7 +78,7 @@ data ClientHandshake = ClientHandshake
}
```
`ServerHandshake` already contains `authPubKey` with the server certificate chain and the signed key for connection encryption and creating a shared secret for denable authorization (with client entity key) and session encryption layer.
`ServerHandshake` already contains `authPubKey` with the router certificate chain and the signed key for connection encryption and creating a shared secret for deniable authorization (with client entity key) and session encryption layer.
`ClientHandshake` contains only ephemeral `authPubKey` to compute a shared secret for session encryption layer, so we need an additional field for an optional client certificate:
@@ -77,9 +86,9 @@ data ClientHandshake = ClientHandshake
serviceCertKey :: Maybe (X.CertificateChain, X.SignedExact X.PubKey)
```
Certificate here defines client identity. The actual key to be used to sign commands is session-scoped, and is signed by the certificate key. In case of notification server it MUST be the same certificate that is used for server TLS connections.
Certificate here defines client identity. The actual key to be used to sign commands is session-scoped, and is signed by the certificate key. In case of notification router it MUST be the same certificate that is used for router TLS connections.
For operators' clients we may optionally include operators' certificate in the chain, and that would allow servers to identify operators if either wants to. This would improve end-user security, as not only the server would validate that its certificate matches the address, but it would also validate that it is operated by SimpleX Chat or by Flux, preventing any server impersonation (e.g., via DNS manipulations) - the client could then report that the files are hosted on SimpleX Chat servers, but then can stop and show additional warning in case certificate does not match the domain - same as the browsers do with CA stores in the client.
For operators' clients we may optionally include operators' certificate in the chain, and that would allow routers to identify operators if either wants to. This would improve end-user security, as not only the router would validate that its certificate matches the address, but it would also validate that it is operated by SimpleX Chat or by Flux, preventing any router impersonation (e.g., via DNS manipulations) - the client could then report that the files are hosted on SimpleX Chat routers, but then can stop and show additional warning in case certificate does not match the domain - same as the browsers do with CA stores in the client.
## Protocol transmissions
@@ -104,9 +113,9 @@ authenticator = queue_authenticator ("0" / "1" service_authenticator)
In case service_authenticator is present, queue_authenticator should authorize over `fingerprint authorized` (concatenation of service identity certificate fingerprint and the rest of the transmission).
All queues created with client key will have to be double-authorized with both the queue key and the client key - both the client and the server would have to maintain this knowledge, whether the queue is associated with the client or not.
All queues created with client key will have to be double-authorized with both the queue key and the client key - both the client and the router would have to maintain this knowledge, whether the queue is associated with the client or not.
Asymmetric retries have to be supported - the first request creating this association may succeed on the server and timeout on the client.
Asymmetric retries have to be supported - the first request creating this association may succeed on the router and timeout on the client.
## Subscription
@@ -118,7 +127,7 @@ The command and response:
SUBS :: Command Recipient -- to enable all client subscriptions, empty entity ID in the transmission, signed by client key - it must be the same as was used in handover subscription signature.
NSUBS :: Command Recipient -- notification subscription
SOK :: Maybe ServiceId -- new subscription response
SOKS :: Int64 -> BrokerMsg -- response from the server, includes the number of subscribed queues
SOKS :: Int64 -> BrokerMsg -- response from the router, includes the number of subscribed queues
ENDS :: Int64 -> BrokerMsg -- when another session subscribes with the same certificate
```
@@ -133,7 +142,7 @@ This was considered to reduce costs for the usual clients to re-subscribe. Curre
For some very busy end-user clients it may help.
Given that server has access to an ephemeral association between recipient client session and queues anyway (even with clients connecting via Tor, unless per-connection transport isolation is used), introducing `sessionPubKey` to allow resubscription to the previously subscribed queues may reduce the traffic. This won't change threat model as the server would only keep this association in memory, and not persist it. Clients on another hand may safely persist this association for fast resubscription on client restarts.
Given that router has access to an ephemeral association between recipient client session and queues anyway (even with clients connecting via Tor, unless per-connection transport isolation is used), introducing `sessionPubKey` to allow resubscription to the previously subscribed queues may reduce the traffic. This won't change threat model as the router would only keep this association in memory, and not persist it. Clients on another hand may safely persist this association for fast resubscription on client restarts.
This is not planned for the forseable future, as migrating to chat relays would solve most of the problem.
@@ -1,10 +1,19 @@
---
Proposed: 2024-02-12
Implemented: ~2024 (SMP v11)
Standardized: 2026-03-10
Protocol: simplex-messaging
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Transmission encryption
## Problems
### Protection of meta-data from sending proxy
The SEND commands and message queue IDs need to be encrypted so that sending proxy cannot see how many queues exist on each server.
The SEND commands and message queue IDs need to be encrypted so that sending proxy cannot see how many queues exist on each router.
Correlation IDs need to be random and can be re-used as nonces so that the destination relay cannot use the increasing correlation IDs that are sent in v6 of the protocol to track the sender.
@@ -24,10 +33,10 @@ encRespTransmission = replyNonce encrypted(respTransmission)
respTransmission = entityId command
```
The keys to encrypt and decrypt both the command and responses would be computed as curve25519 from the key sent together with command and server session key. For the requests, the nonce has to be random and sent outside of the encrypted envelopt, but for the response respNonce would be taken from inside of the encrypted envelope and it would also be used for correlating commands and responses. This way the attacker who could compromise TLS would not be able to correlate the commands and responses, and also observe entity IDs.
The keys to encrypt and decrypt both the command and responses would be computed as curve25519 from the key sent together with command and router session key. For the requests, the nonce has to be random and sent outside of the encrypted envelope, but for the response respNonce would be taken from inside of the encrypted envelope and it would also be used for correlating commands and responses. This way the attacker who could compromise TLS would not be able to correlate the commands and responses, and also observe entity IDs.
2. The remaining question is to how encrypt and decrypt messages delivered not in response to the commands.
The possible options are:
- restore client session key only for that purpose, but do not forward this key to the destination proxy for sent messages. Then the messages can be sent with a random replyNonce and the key would be computed from session keys. The advantage here is that we won't need to parameterize handles as both client and server would have session keys. The downside that we would have to either somehow differentiate messages and responses, either by some flag that would allow some correlation or just by the absense of replyNonce in the lookup map - that is if the client can find replyNonce, it would use the associated key to decrypt, and if not it would use session key.
- use the same key that was sent with SUB or ACK command. This is much more complex, and would only have some upside if we were to introduce receiving proxies (to conceal transport sessions from the receiving relays for the recipients).
- use the same key that was sent with SUB or ACK command. This is much more complex, and would only have some upside if we were to introduce receiving proxies (to conceal transport sessions from the receiving routers for the recipients).
@@ -1,8 +1,17 @@
---
Proposed: 2024-03-20
Implemented: ~2024
Standardized: 2026-03-10
Protocol: simplex-messaging
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Relay metadata and SimpleX network decentralization
## Problem
Currently, the clients configure/choose which servers to use, but they cannot see who operates them, in which geography and hosting provider, what is the server source code (in case it was modified from the reference implementation we provide) and also any administrative and feedback contacts.
Currently, the clients configure/choose which routers to use, but they cannot see who operates them, in which geography and hosting provider, what is the router source code (in case it was modified from the reference implementation we provide) and also any administrative and feedback contacts.
Further, we currently use simplex.chat domain to host group links, and as diversity of the groups grows it is beginning to require managing feedback from the users about groups. It is important that this feedback is directed to relay owners and not to us, in case they are not our relays, as we are simply providing software here.
@@ -21,28 +30,28 @@ While this document is not the end of the journey to decentralize the network, i
The proposed solution consists of two parts:
- communicate server metadata via protocol, so it can be observed by the clients.
- communicate router metadata via protocol, so it can be observed by the clients.
- create home page for the relays, with all the same metadata.
- create invitation and address links in the same domain name as the relay.
The latter point is important so it is clear to the users who operates and owns the relay and where the access point to the content or group is hosted. Even though simplex.chat domain is never accessed by the app, and the meaningful part of the address is never sent to the page hosting server, it creates an impression of centralization, and some dependency on simplex.chat domain for anything other that showing the link QR code.
The latter point is important so it is clear to the users who operates and owns the relay and where the access point to the content or group is hosted. Even though simplex.chat domain is never accessed by the app, and the meaningful part of the address is never sent to the page hosting router, it creates an impression of centralization, and some dependency on simplex.chat domain for anything other that showing the link QR code.
Moving invitation links to the domain of the relay (primary relay, in case the link has redundancy) will both clarify relay ownership, solve the incorrect mis-perception of centralization, remove the dependency on simplex-chat domain without any user effort, and provides the means to submit content complaints to the relay operators (should they wish to receive them, which seems reasonable for large public relays, but may be unnecessary for private relays where unidentified parties cannot create links).
## Solution details
Extend server INI file with information section:
Extend router INI file with information section:
```
[INFORMATION]
# Please note that under AGPLv3 license conditions you MUST make
# any source code modifications available to the end users of the server.
# any source code modifications available to the end users of the router.
# LICENSE: https://github.com/simplex-chat/simplexmq/blob/stable/LICENSE
# Not doing so would constitute a license violation.
# Declaring an incorrect information here amounts to a fraud.
# The license holders reserve the right to prosecute missing or incorrect
# information about the server source code to the fullest extent permitted by the law.
# The server will show warning on start if this field is absent
# The router will show warning on start if this field is absent
# and will not launch from v6.0 until this field is added.
# If any other information field is present, source code property also MUST be present.
source_code: https://github.com/simplex-chat/simplexmq
@@ -69,13 +78,13 @@ hosting: Linode / Akamai Inc.
hosting_country: US
```
Server home page would show whether queue creation is allowed and/or password protected, server retention policy (e.g., preserve messages on restart or not, and persist connections or not).
Router home page would show whether queue creation is allowed and/or password protected, router retention policy (e.g., preserve messages on restart or not, and persist connections or not).
Server queue address/contact pages will optionally, provide the UI to submit feedback, comments and complaints directly from the web page (not an MVP, initially we would simply show addresses for feedback, and, probably, create link that opens in the app with pre-populated message, and we could also use this addresses defined in server meta-data to submit feedback from inside of the app - it's also out of MVP scope).
Router queue address/contact pages will optionally, provide the UI to submit feedback, comments and complaints directly from the web page (not an MVP, initially we would simply show addresses for feedback, and, probably, create link that opens in the app with pre-populated message, and we could also use this addresses defined in router meta-data to submit feedback from inside of the app - it's also out of MVP scope).
If server is available on .onion address, the web pages would show "open via .onion" in Tor browser.
If router is available on .onion address, the web pages would show "open via .onion" in Tor browser.
Extend server handshake header with these information fields:
Extend router handshake header with these information fields:
```haskell
data ServerHandshake = ServerHandshake
@@ -93,13 +102,13 @@ data ServerInformation = ServerInformation
info :: ServerPublicInfo
}
-- based on server configuration
-- based on router configuration
data ServerPublicConfig = ServerPublicConfig
{ persistence :: SMPServerPersistenceMode,
messageExpiration :: Int,
statsEnabled :: Bool,
newQueuesAllowed :: Bool,
basicAuthEnabled :: Bool -- server is private if enabled
basicAuthEnabled :: Bool -- router is private if enabled
}
-- based on INFORMATION section of INI file
@@ -127,4 +136,4 @@ data ServerContactAddress = ServerContactAddress
}
```
This extended server information will be stored in the chat database every time it changes and shown in the UI of the server configuration.
This extended router information will be stored in the chat database every time it changes and shown in the UI of the router configuration.
@@ -1,3 +1,12 @@
---
Proposed: 2024-06-21
Implemented: ~2025 (SMP v15)
Standardized: 2026-03-10
Protocol: simplex-messaging + agent-protocol
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Short invitation links
## Problem
@@ -14,7 +23,7 @@ Additionally, if we store short links, they can also include chat preferences an
MITM-resistant link shortening.
Instead of generating the random address that would resolve into the link - doing so would create the possibility of MITM by the server hosting this link - we can use private key as the link ID that will be passed to the accepting party, and the hash of the public key as ID for the server - the accepting party would present this key itself as ID and it will also be used for server to client encryption (see Protocol below). HKDF will be used to derive symmetric key from private key and used in secret_box together with random nonce (to allow replacing data with the same key but with a different nonce - nonce will be sent to the server too). secret_box construction is authenticated encryption, so it would protect from MITM.
Instead of generating the random address that would resolve into the link - doing so would create the possibility of MITM by the router hosting this link - we can use private key as the link ID that will be passed to the accepting party, and the hash of the public key as ID for the router - the accepting party would present this key itself as ID and it will also be used for router to client encryption (see Protocol below). HKDF will be used to derive symmetric key from private key and used in secret_box together with random nonce (to allow replacing data with the same key but with a different nonce - nonce will be sent to the router too). secret_box construction is authenticated encryption, so it would protect from MITM.
The proposed syntax:
@@ -29,7 +38,7 @@ srvHosts = <hostname> ["," srvHosts] ; RFC1123, RFC5891
linkHash = <base64url encoded SHA256 or SHA512 hash of the original link>
```
If SMP server supports pages, its name can be used as clientAppServer, without repeating it after #, for a shorter link.
If SMP router supports pages, its name can be used as clientAppServer, without repeating it after #, for a shorter link.
Example link:
@@ -40,12 +49,12 @@ https://simplex.chat/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU=@smp8.
This link has the length of ~136 characters (256 bits), which is shorter than the full contact address (~310 characters) and much shorter than invitation links (~528 characters) even without post-quantum keys added to them.
This size can be further reduced by
- use server domain in the link.
- do not include onion address, as the connection happens via proxy anyway, if it's untrusted server.
- not pinning server TLS certificate - the downside here is that while the attack that compromises TLS will not be able to substitute the link (because it's hash will not match), it will be able to intercept and to block it.
- use router domain in the link.
- do not include onion address, as the connection happens via proxy anyway, if it's untrusted router.
- not pinning router TLS certificate - the downside here is that while the attack that compromises TLS will not be able to substitute the link (because it's hash will not match), it will be able to intercept and to block it.
- using shorter hash, e.g. SHA128 - reducing the collision resistance.
If the server is known, the client could use it's hash and onion address, otherwise it could trust the proxy to use any existing session with the same hostname or to accept the risk of interception - given that there is no risk of substitution.
If the router is known, the client could use its hash and onion address, otherwise it could trust the proxy to use any existing session with the same hostname or to accept the risk of interception - given that there is no risk of substitution.
With the first two of these "improvements" the link could be ~122 characters:
@@ -59,13 +68,13 @@ If onion address is preserved the link will be ~184 characters (won't fit in Twi
https://smp8.simplex.im/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU@beccx4yfxxbvyhqypaavemqurytl6hozr47wfc7uuecacjqdvwpw2xid.onion/abcdefghij0123456789abcdefghij0123456789abc
```
If we implement it, the request to resolve the link would be made via proxied SMP command (to avoid the direct connection between the client and the recipient's server).
If we implement it, the request to resolve the link would be made via proxied SMP command (to avoid the direct connection between the client and the recipient's router).
Pros:
- a bit shorter link.
- possibility to include post-quantum keys into the full link keeping the same shortened link size.
- possibility to include chat profile of contact or group, and preferences, for a much better connection experience, and to show this information when the link sent in the conversation (clients can resolve them automatically, without connecting - it can be resolved by the sending clients).
- server will not have access to the link.
- router will not have access to the link.
Cons:
- protocol complexity.
@@ -75,7 +84,7 @@ Pros are a huge improvement of UX of connecting both within and from outside of
## Protocol
To support short links, the SMP servers would provide a simple key-value store enabled by three additional commands: `WRT`, `CLR` and `READ`
To support short links, the SMP routers would provide a simple key-value store enabled by three additional commands: `WRT`, `CLR` and `READ`
`WRT` command is used to store and to update values in the store. The size of the value is limited by the same size as sent messages (or, possibly, smaller - as connection information size used in confirmation messages) - the clients would use this fixed size irrespective of the content. `WRT` command will be sent with the data blob ID in the transaction entityId field, public authorization key used to authorize `WRT` and `CLR` commands (subsequent WRT commands to the existing key must use the same key), and the data blob.
@@ -89,22 +98,22 @@ To support short links, the SMP servers would provide a simple key-value store e
- the data blob owner generates X25519 key pair: `(k, pk)`.
- private key `pk` will be included in the short link shared with the other party (only base64url encoded key bytes, not X509 encoding).
- `HKDF(pk)` will be used to encrypt the link data with secret_box before storing it on the server.
- `HKDF(pk)` will be used to encrypt the link data with secret_box before storing it on the router.
- the hash of public key `sha256(k)` will be used as ID by the owner to store and to remove the data blob (`WRT` and `CLR` commands).
**Retrieve data blob**
- the sender uses the public key `k` derived from the private key `pk` included in the link as entity ID to retrieve data blob (the server will compute the ID used by the owner as `sha256(k)` and will be able to look it up). This provides the quality that the traffic of the parties has no shared IDs inside TLS. It also means that unlike message queue creation, the ID to retrieve the blob was never sent to the blob creator, and also is not known to the server in advance (the second part is only an observation, in itself it does not increase security, as server has access to an encrypted blob anyway).
- the sender uses the public key `k` derived from the private key `pk` included in the link as entity ID to retrieve data blob (the router will compute the ID used by the owner as `sha256(k)` and will be able to look it up). This provides the quality that the traffic of the parties has no shared IDs inside TLS. It also means that unlike message queue creation, the ID to retrieve the blob was never sent to the blob creator, and also is not known to the router in advance (the second part is only an observation, in itself it does not increase security, as router has access to an encrypted blob anyway).
- note that the sender does not authorize the request to retrieve the blob, as it would not increase security unless a different key is used to authorize, and adding a key would increase link size.
- server session keys with the sender will be `(sk, spk)`, where `sk` is public key shared with the sender during session handshake, and `spk` is the private key known only to the server.
- this public key `k` will also be combined with server session key `spk` using `dh(k, spk)` to encrypt the response, so that there is no ciphertext in common in sent and received traffic for these blobs. Correlation ID will be used as a nonce for this encryption.
- router session keys with the sender will be `(sk, spk)`, where `sk` is public key shared with the sender during session handshake, and `spk` is the private key known only to the router.
- this public key `k` will also be combined with router session key `spk` using `dh(k, spk)` to encrypt the response, so that there is no ciphertext in common in sent and received traffic for these blobs. Correlation ID will be used as a nonce for this encryption.
- having received the blob, the client can now decrypt it using secret_box with `HKDF(pk)`.
Using the same key as ID for the request, and also to additionally encrypt the response allows to use a single key in the link, without increasing the link size.
## Threat model
**Compromised SMP server**
**Compromised SMP router**
can:
- delete link data.
@@ -1,3 +1,12 @@
---
Proposed: 2024-09-09
Implemented: ~2025 (SMP v15)
Standardized: 2026-03-10
Protocol: simplex-messaging + agent-protocol
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Blob extensions for SMP queues
Evolution of the design for short links, see [here](./2024-06-21-short-links.md) and [here](./2024-09-05-queue-storage.md).
@@ -11,13 +20,13 @@ Allow storing extended information with SMP queues to improve UX and security of
## Design
1. Queue creation/update date is already added to server persistence, allowing to expire queues and blobs, depending on their usage.
1. Queue creation/update date is already added to router persistence, allowing to expire queues and blobs, depending on their usage.
2. Add "queue type" metadata to NEW command to indicate whether messaging queue is used as public address or as messaging queue (see previous docs on why it doesn't change threat model). While at the moment it would match sndSecure flag there may be future scenarios when they diverge. Initially only "invitation" and "contact" types will be supported.
3. Prohibit sndSecure flag for "contact" queues, prohibit securing contact queues.
4. Add "queue blobs" to NEW command:
- blob0: ratchetKeys up to N0 bytes - priority 0, can't be removed by the server, only in "invitation"
- blob1: PQ key up to N1 bytes - priority 1, can be removed by the server, only used in "invitation"
- blob2: Application data up to N2 bytes - priority 2, can be removed by the server.
- blob0: ratchetKeys up to N0 bytes - priority 0, can't be removed by the router, only in "invitation"
- blob1: PQ key up to N1 bytes - priority 1, can be removed by the router, only used in "invitation"
- blob2: Application data up to N2 bytes - priority 2, can be removed by the router.
5. Add linkId to NEW command
6. linkId and blobs will be removed when queue is secured.
7. Add recipient command to remove/upsert blob2 for contact queues.
@@ -28,7 +37,7 @@ Allow storing extended information with SMP queues to improve UX and security of
### Creating a queue:
The queue owner:
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now. `sk` and `dhk` will be sent in NEW command.
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the router, same as now. `sk` and `dhk` will be sent in NEW command.
- generates X25519 key pair `(k, pk)` to use with the accepting party to encrypt queue messages.
- derives from `k` using HKDF:
- symmetric key `bk` for authenticated encryption of blobs.
@@ -73,7 +82,7 @@ Response to GET:
blobs = %s"BLOB" senderId [ "0" blob0 ] [ "1" blob1 ] [ "2" blob2 ]
```
As blobs are retrieved using a separate linkId, once blobs are removed it will be impossible to find senderId from short link - it is a threat model improvement. Once server storage is compacted, it will be impossible to find queue related to the link even with the access to server data (unless server preserves the data).
As blobs are retrieved using a separate linkId, once blobs are removed it will be impossible to find senderId from short link - it is a threat model improvement. Once router storage is compacted, it will be impossible to find queue related to the link even with the access to router data (unless router preserves the data).
### Possible privacy improvement
@@ -1,3 +1,12 @@
---
Proposed: 2025-03-16
Implemented: ~2025 (SMP v15)
Standardized: 2026-03-10
Protocol: simplex-messaging + agent-protocol
---
> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior.
# Protocol changes for creating and connecting to SMP queues
## Problems
@@ -19,18 +28,18 @@ Simply designating queue types would allow to use this information to decide for
We want to achieve these objectives for short links and associated queue data:
1. no possibility to provide incorrect SenderId inside link data (e.g. from another queue).
2. link data cannot be accessed by the server unless it has the link.
3. prevent MITM attack by the server, including the server that obtained the link.
2. link data cannot be accessed by the router unless it has the link.
3. prevent MITM attack by the router, including the router that obtained the link.
4. prevent changing of connection request by the user (to prevent MITM via break-in attack in the originating client).
5. for one-time links, prevent accessing link data by link observers who did not compromise the server.
5. for one-time links, prevent accessing link data by link observers who did not compromise the router.
6. allow changing the user-defined part of link data.
7. avoid changing the link when user-defined part of link data changes, while preventing MITM attack by the server on user-defined part, even if it has the link.
8. retain the quality that it is impossible to check the existence of secured queue from having any of its temporary visible IDs (sender ID and link ID in 1-time invitations) - it requires that these IDs remain server-generated (contrary to the previous RFCs).
7. avoid changing the link when user-defined part of link data changes, while preventing MITM attack by the router on user-defined part, even if it has the link.
8. retain the quality that it is impossible to check the existence of secured queue from having any of its temporary visible IDs (sender ID and link ID in 1-time invitations) - it requires that these IDs remain router-generated (contrary to the previous RFCs).
To achieve these objectives the queue data will include fixed (immutable) and user-defined (mutable) parts.
Fixed part would include:
- full connection request (the current long link with all keys, including PQ keys). This includes SenderId that must match server response.
- full connection request (the current long link with all keys, including PQ keys). This includes SenderId that must match router response.
- public signature key to verify mutable part of link data.
Signed mutable part would include:
@@ -41,7 +50,7 @@ The link itself should include both the key and auth tag from the encryption of
## Solution
Current NEW and NKEY commands:
Current NEW and NKEY commands (code identifiers like `QueueIdsKeys` are Haskell type names):
```haskell
NEW :: RcvPublicAuthKey -> RcvPublicDhKey -> Maybe BasicAuth -> SubscriptionMode -> SenderCanSecure -> Command Recipient
@@ -76,8 +85,8 @@ data QueueReqData
| QRContact (Maybe (LinkId, (SenderId, QueueLinkData)))
-- SenderId should be computed client-side as the first 24 bytes of sha3-384(correlation_id),
-- The server must verify it and reject if it is not.
-- It allows to include sender ID inside encrypted associated link data as part of full connection URI without requesting it from the server, but prevents checking if a given sender ID exists (queue creation would fail for a duplicate sender ID), as sha3-384 derivation is not reversible.
-- The router must verify it and reject if it is not.
-- It allows to include sender ID inside encrypted associated link data as part of full connection URI without requesting it from the router, but prevents checking if a given sender ID exists (queue creation would fail for a duplicate sender ID), as sha3-384 derivation is not reversible.
type QueueLinkData = (EncFixedLinkData, EncUserDataBytes)
type EncFixedLinkData = ByteString
@@ -86,7 +95,7 @@ type EncUserDataBytes = ByteString
-- We need to use binary encoding for ConnectionRequestUri to reduce its size
-- The clients would reject changed immutable data and
-- ConnectionRequestUri where server or SenderId of the queue do not match.
-- ConnectionRequestUri where router or SenderId of the queue do not match.
data FixedLinkData c = FixedLinkData
{ agentVRange :: VersionRangeSMPA,
rootKey :: C.PublicKeyEd25519,
@@ -110,11 +119,11 @@ newtype UserLinkData = UserLinkData ByteString
-- | Updated queue IDs and keys, returned in IDS response
data QueueIdsKeys = QIK
{ rcvId :: RecipientId, -- server-generated
sndId :: SenderId, -- server-generated
{ rcvId :: RecipientId, -- router-generated
sndId :: SenderId, -- router-generated
rcvPublicDhKey :: RcvPublicDhKey,
sndSecure :: SenderCanSecure, -- possibly, can be removed? or implied?
linkId :: Maybe LinkId -- server-generated
linkId :: Maybe LinkId -- router-generated
}
```
@@ -149,31 +158,31 @@ LGET :: Command Sender
LNK :: SenderId -> QueueLinkData -> BrokerMsg
```
To both include sender_id into the full link before the server response, and to prevent "oracle attack" when a failure to create the queue with the supplied `sender_id` can be used as a proof of queue existence, it is proposed that `sender_id` is computed client-side as the first 24 bytes of 48 in `sha3-384(correlation_id)` and validated server-side, where `corelation_id` is the transmission correlation ID.
To both include sender_id into the full link before the router response, and to prevent "oracle attack" when a failure to create the queue with the supplied `sender_id` can be used as a proof of queue existence, it is proposed that `sender_id` is computed client-side as the first 24 bytes of 48 in `sha3-384(correlation_id)` and validated router-side, where `corelation_id` is the transmission correlation ID.
To allow retries, every time the command is sent a new random `correlation_id` and new `sender_id` (and for contact queue, also `link_id`, which would be random as it is derived from hash of fixed link data that includes a random signature key) should be used on each attempt, because other IDs would be generated randomly on the server, and in case the previous command succeeded on the server but failed to be communicated to the client, the retry will fail if the same ID is used.
To allow retries, every time the command is sent a new random `correlation_id` and new `sender_id` (and for contact queue, also `link_id`, which would be random as it is derived from hash of fixed link data that includes a random signature key) should be used on each attempt, because other IDs would be generated randomly on the router, and in case the previous command succeeded on the router but failed to be communicated to the client, the retry will fail if the same ID is used.
Alternative solutions that would allow retries that were considered and rejected:
- additional request to save queue data, after `sender_id` is returned by the server. The scenarios that require short links are interactive - creating user addresses and 1-time invitations - so making two requests instead of one would make the UX worse.
- include empty sender_id in the immutable data and have it replaced by the accepting party with `sender_id` received in `LINK` response - both a weird design, and might create possibility for some attacks via server, especially for contact addresses.
- additional request to save queue data, after `sender_id` is returned by the router. The scenarios that require short links are interactive - creating user addresses and 1-time invitations - so making two requests instead of one would make the UX worse.
- include empty sender_id in the immutable data and have it replaced by the accepting party with `sender_id` received in `LINK` response - both a weird design, and might create possibility for some attacks via router, especially for contact addresses.
- making NEW commands idempotent. Doing it would require generating all IDs client-side, not only `sender_id`. It increases complexity, and it is not really necessary as the only scenarios when retries are needed are async NEW commands, that do not require short links. For future short links of chat relays the retries are much less likely, as chat relays will have good network connections.
## Algorithm to prepare and to interpret queue link data.
For contact addresses this approach follows the design proposed in [Short links](./2024-06-21-short-links.md) RFC - when link id is derived from the same random binary as key. For 1-time invitations link ID is independent and server-generated, to prevent existence checks (oracle attack).
For contact addresses this approach follows the design proposed in [Short links](./2024-06-21-short-links.md) RFC - when link id is derived from the same random binary as key. For 1-time invitations link ID is independent and router-generated, to prevent existence checks (oracle attack).
This scheme results in 32 byte binary size for contact addresses and 56 bytes for 1-time invitation links.
For fixed link data.
1. Generate random `nonce` (also used as a correlation ID for server command) and signature key (public `rootKey` included in fixed data).
1. Generate random `nonce` (also used as a correlation ID for router command) and signature key (public `rootKey` included in fixed data).
2. Compute sender ID from `nonce` as the first 24 bytes of sha3-384 of `nonce`.
3. Generate other keys for queue address, including queue e2e encryption keys and double ratchet connection e2e encryption keys.
4. Construct the full connection address to be included in fixed data.
5. `link_key = SHA3-256(fixed_data)` - used as part of the link, and to derive the key to encrypt content.
6. HKDF:
1) contact address: `(link_id, key) = HKDF(link_key, 56 bytes)`.
2) 1-time invitation: `key = HKDF(link_key, 32 bytes)`, `link-id` - server-generated.
2) 1-time invitation: `key = HKDF(link_key, 32 bytes)`, `link-id` - router-generated.
7. Encrypt: `(ct1, tag1) = secret_box(fixed_data, key, nonce1)`, where `nonce1` is a random nonce
5. Store: `(nonce1, ct1, tag1)` stored as fixed link data.
@@ -202,7 +211,7 @@ While using content hash as encryption key is unconventional, it is not complete
## Threat model
**Compromised SMP server**
**Compromised SMP router**
can:
- delete link data.
@@ -223,22 +232,22 @@ cannot:
- undetectably check the existence of messaging queue or 1-time link (objective 8).
- replace or delete the link data.
**Queue owner who did not compromise the server**:
**Queue owner who did not compromise the router**:
cannot:
- redirect connecting user to another queue, on the same or on another server (objective 1).
- redirect connecting user to another queue, on the same or on another router (objective 1).
- replace connection request in the link (objective 4).
## Correlation of design objectives with design elements
1. The presence of `SenderId` in `LNK` response from the server.
1. The presence of `SenderId` in `LNK` response from the router.
2. Encryption of link data with crypto_box.
3. Deriving encryption key from the hash of fixed data prevents it being modified by the server - any change would be detected and rejected by the client, as the hash of fixed data won't match the link. Signature verification with the key from fixed data, and signing of mutable data prevents server modification of mutable data.
4. No server command to change fixed data once it's set. Also, changing fixed data would require changing the link.
3. Deriving encryption key from the hash of fixed data prevents it being modified by the router - any change would be detected and rejected by the client, as the hash of fixed data won't match the link. Signature verification with the key from fixed data, and signing of mutable data prevents router modification of mutable data.
4. No router command to change fixed data once it's set. Also, changing fixed data would require changing the link.
5. 1-time link data can only be accessed with `LKEY` command, that while allows retries to mitigate network failures, will require the same key for retries.
6. `LSET` command.
7. The link is derived from fixed data only, so it does not change when mutable link data changes. Mutable part is signed preventing server MITM attacks.
8. SenderId is derived from request correlation ID, so it cannot be arbitrary defined to check existence of some known queue. LinkId for 1-time invitation is generated server-side, so it cannot be provided by the client when creating the queues to check if these IDs are used.
7. The link is derived from fixed data only, so it does not change when mutable link data changes. Mutable part is signed preventing router MITM attacks.
8. SenderId is derived from request correlation ID, so it cannot be arbitrary defined to check existence of some known queue. LinkId for 1-time invitation is generated router-side, so it cannot be provided by the client when creating the queues to check if these IDs are used.
## Syntax for short links
@@ -257,34 +266,34 @@ contactLink = <base64url(linkKey)> ; 32 bytes / 43 base64 encoded characters
param = hostsParam / portParam / certHashParam
hostsParam = %s"h=" host *("," host) ; additional hostnames, e.g. onion
portParam = %s"p=" 1*DIGIT ; server port
certHashParam = %s"c=" <base64url(server offline certificate fingerprint)>
portParam = %s"p=" 1*DIGIT ; router port
certHashParam = %s"c=" <base64url(router offline certificate fingerprint)>
```
To have shorter links fingerprint and additional server hostnames do not need to be specified for pre-configured servers, even if they are disabled - they can be used from the client code. Any user defined servers will require including additional hosts and server fingerprint.
To have shorter links fingerprint and additional router hostnames do not need to be specified for pre-configured routers, even if they are disabled - they can be used from the client code. Any user defined routers will require including additional hosts and router fingerprint.
Example one-time link for preset server (104 characters):
Example one-time link for preset router (104 characters):
```
https://smp12.simplex.im/i#abcdefghij0123456789abcdefghij01/23456789abcdefghij0123456789abcdefghij01234
```
Example contact link for preset server (71 characters):
Example contact link for preset router (71 characters):
```
https://smp12.simplex.im/c#abcdefghij0123456789abcdefghij0123456789abc
```
Example contact link for user-defined server (with fingerprint, but without onion hostname - 117 characters):
Example contact link for user-defined router (with fingerprint, but without onion hostname - 117 characters):
```
https://smp1.example.com/c#abcdefghij0123456789abcdefghij0123456789abc?c=0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU
```
Example contact link for user-defined server (with fingerprint ant onion hostname - 182 characters):
Example contact link for user-defined router (with fingerprint and onion hostname - 182 characters):
```
https://smp1.example.com/c#abcdefghij0123456789abcdefghij0123456789abc?c=0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU&h=beccx4yfxxbvyhqypaavemqurytl6hozr47wfc7uuecacjqdvwpw2xid.onion
```
For the links to work in the browser the servers must provide server pages.
For the links to work in the browser the routers must provide router pages.
+6 -6
View File
@@ -67,14 +67,14 @@ if [ ! -f "${confd}/smp-server.ini" ]; then
# Fix path to certificates
if [ -n "${WEB_MANUAL}" ]; then
sed -i -e 's|^[^#]*https: |#&|' \
-e 's|^[^#]*cert: |#&|' \
-e 's|^[^#]*key: |#&|' \
-e 's|^port:.*|port: 5223|' \
sed -i -e 's|^[^#]*https = |#&|' \
-e 's|^[^#]*cert = |#&|' \
-e 's|^[^#]*key = |#&|' \
-e 's|^port = .*|port = 5223|' \
"${confd}/smp-server.ini"
else
sed -i -e "s|cert: /etc/opt/simplex/web.crt|cert: $cert_path/$ADDR.crt|" \
-e "s|key: /etc/opt/simplex/web.key|key: $cert_path/$ADDR.key|" \
sed -i -e "s|cert = /etc/opt/simplex/web.crt|cert = $cert_path/$ADDR.crt|" \
-e "s|key = /etc/opt/simplex/web.key|key = $cert_path/$ADDR.key|" \
"${confd}/smp-server.ini"
fi
fi
+1 -1
View File
@@ -76,7 +76,7 @@ if [ ! -f "${confd}/file-server.ini" ]; then
# Optionally, set password
if [ -n "${PASS}" ]; then
sed -i -e "/^# create_password:/a create_password: $PASS" \
sed -i -e "/^# create_password =/a create_password = $PASS" \
"${confd}/file-server.ini"
fi
fi
+2 -2
View File
@@ -237,11 +237,11 @@ checks() {
exit 1
fi
mkdir -p "$path_conf_info" "$path_tmp_bin"
check_versions
check_distro
mkdir -p $path_conf_info $path_tmp_bin
return 0
}
+22
View File
@@ -0,0 +1,22 @@
# ============================================================================
# Required settings — the stack will not start without these.
# ============================================================================
# Ethereum network: mainnet (the SNRC `.testing` contracts live on mainnet)
# or holesky (test). Mainnet full sync needs ~1 day and ~1.2 TB NVMe.
NETWORK=mainnet
# Beacon checkpoint-sync URL — used ONCE on first sync. Must expose the heavy
# /eth/v2/debug/beacon/states/finalized endpoint (generic beacon APIs do not;
# use a dedicated checkpoint provider). List: https://eth-clients.github.io/checkpoint-sync-endpoints/
# mainnet: https://mainnet-checkpoint-sync.attestant.io (also beaconstate.info, sync-mainnet.beaconcha.in)
# holesky: https://checkpoint-sync.holesky.ethpandaops.io
TRUSTED_NODE_URL=https://mainnet-checkpoint-sync.attestant.io
# ============================================================================
# Optional overrides — sensible defaults are baked into docker-compose.yml,
# so leave these commented unless you need to change them.
# ============================================================================
# Nimbus NAT (default: any). For a stable public node set an explicit IP:
# NAT=extip:1.2.3.4 # your public IPv4: curl -s ifconfig.me
+146
View File
@@ -0,0 +1,146 @@
# Self-hosted SNRC stack
One `docker compose up` runs the self-hosted SimpleX Namespace (SNRC) backend
against **Ethereum mainnet** (where the `.testing` contracts live):
| # | Component | What it does |
|---|---|---|
| 1 | **reth + nimbus** | self-hosted Ethereum node (`--minimal` — enough for the resolver's `eth_call` at chain head) |
| 2 | **resolver** | the REST resolver the smp-server's `[NAMES]` role queries (`snrc-resolve.py`) |
## Requirements
- **Docker** + Compose v2.
- **≥ 300 GB NVMe SSD** for `reth --minimal` (~260 GB on mainnet; TLC, not QLC
— QLC stalls during sync) + **32 GB RAM**, fast multi-core CPU.
- **~1 day** for the initial reth sync. The resolver returns errors until reth
has caught up — that's expected.
- Firewall: open p2p ports `30303` (tcp/udp) and `9000` (tcp/udp).
## 1. Configure
Edit `.env` — the defaults work as-is; override only if needed:
```sh
NETWORK=mainnet # default
TRUSTED_NODE_URL=https://mainnet-checkpoint-sync.attestant.io # default
```
Everything else (NAT) has a working default baked into `docker-compose.yml`;
uncomment the hints in `.env` only to override.
## 2. Run
```sh
cd scripts/resolver
docker compose up -d
docker compose logs -f reth resolver
```
`depends_on` handles ordering automatically (start node → start resolver).
## 3. Wait for the node to sync
```sh
docker compose logs --tail=20 reth
```
This is the long pole (~1 day on mainnet). Until reth is synced the resolver
returns `502`.
## Verify
Run these once the stack is up (the node-dependent ones pass after sync):
**1. reth is reachable and reporting a block:**
```sh
curl -s -X POST http://127.0.0.1:8545 \
-H 'content-type: application/json' \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq
```
**2. resolver is healthy:**
```sh
curl -s http://127.0.0.1:8000/health | jq
# → {"ok": true, "rpc": "http://reth:8545", "registries": {"testing": "0x…", "simplex": ""}}
```
**3. resolver resolves a live name** (`foobar.testing` is a populated test name):
```sh
curl -s http://127.0.0.1:8000/resolve/foobar.testing | jq
# → {"name":"foobar.testing","nickname":"Foo","simplexContact":["https://smp16.simplex.im/a#…"], … }
```
**Wire your smp-server:** in its `[NAMES]` section set
`resolver_endpoint: http://127.0.0.1:8000` (no auth needed for loopback).
## Ports (all loopback unless noted)
| Service | Host | Purpose |
|---|---|---|
| reth JSON-RPC | `127.0.0.1:8545` | smp-server RPC |
| reth p2p | `:30303` tcp/udp | Ethereum sync (open on firewall) |
| nimbus p2p | `:9000` tcp/udp | beacon sync (open on firewall) |
| nimbus REST | `127.0.0.1:5052` | beacon API |
| **resolver** | `127.0.0.1:8000` | SNRC REST (`/resolve`, `/health`) |
## Caveats
- **All images track `:latest`** (reth, nimbus) — you get upstream fixes on each
`docker compose pull`; re-run the verify checks after pulling.
- All ports bind to loopback; expose only what you put behind a TLS reverse proxy.
## Teardown
```sh
docker compose down # stop, keep all state
docker compose down -v # also wipe volumes → full re-sync
```
`down -v` wipes the chain data (full re-sync on the next `up`).
---
## Resolver API reference
The resolver (`snrc-resolve.py`, host `127.0.0.1:8000`) is also runnable
standalone for local dev (no Docker), via [`uv`](https://docs.astral.sh/uv/):
```sh
uv run scripts/resolver/service/snrc-resolve.py # defaults to local reth + mainnet .testing
```
### Response shape
```jsonc
{
"name": "foobar.testing",
"nickname": "Foo", "website": "https://foo.bar", "location": "",
"simplexContact": ["https://smp16.simplex.im/a#…", "https://smp11…"], // primary first, fallbacks after
"simplexChannel": [],
"eth": null, "btc": "bc1q…", "xmr": "4ANz…", "dot": "139G…",
"owner": "0xd83b…", "resolver": "0x80fa…"
}
```
`simplexContact`/`simplexChannel` are arrays (a name can advertise multiple SMP
servers; clients try them in order). On-chain they're a single comma-separated
text record; the resolver splits/trims/drops-empties. Address encodings are
canonical per chain (EIP-55 / bech32 / SS58 / Monero-base58). Subnames work
identically (`bar.foobar.testing`).
### Status codes
| Status | Meaning |
|---|---|
| 200 | resolved |
| 400 | TLD not configured, or not a fully-qualified name |
| 404 | name has no resolver set on the registry |
| 502 | upstream RPC error / reth not synced |
### Configuring registries
Defaults to mainnet `.testing` (`0x03f438…`); `.simplex` is unset until
deployed. Override per TLD via env on the `resolver` service in
`docker-compose.yml` (`SNRC_REGISTRY_TESTING` / `SNRC_REGISTRY_SIMPLEX`), or as
env vars for the standalone script.
+160
View File
@@ -0,0 +1,160 @@
services:
# One-shot setup (runs as root): generates /jwt/jwt.hex and chowns the
# nimbus-data volume to UID 1000 (the user Nimbus runs as inside its image).
# Without this chown Nimbus gets "Permission denied" on its data dir
# because docker creates fresh named volumes owned by root.
init:
image: alpine:latest
volumes:
- jwt:/jwt
- nimbus-data:/nimbus-data
command: >
sh -c '
set -e;
if [ ! -f /jwt/jwt.hex ]; then
apk add --no-cache openssl >/dev/null;
openssl rand -hex 32 | tr -d "\n" > /jwt/jwt.hex;
chmod 644 /jwt/jwt.hex;
echo "Generated /jwt/jwt.hex";
else
echo "jwt.hex already exists";
fi;
chown 1000:1000 /nimbus-data;
echo "Chowned /nimbus-data to 1000:1000";
'
restart: "no"
# One-shot: fetches a recent finalised checkpoint into the Nimbus data dir
# using the trustedNodeSync subcommand. Skipped if the data dir is already
# initialised, so subsequent compose-ups are no-ops.
nimbus-checkpoint-sync:
image: statusim/nimbus-eth2:multiarch-latest
depends_on:
init:
condition: service_completed_successfully
volumes:
- nimbus-data:/home/user/nimbus-eth2/build/data
entrypoint:
- sh
- -c
- |
if [ -d /home/user/nimbus-eth2/build/data/${NETWORK}/db ]; then
echo "Nimbus data dir already initialised — skipping checkpoint sync";
exit 0;
fi;
/home/user/nimbus-eth2/build/nimbus_beacon_node trustedNodeSync \
--network=${NETWORK} \
--data-dir=/home/user/nimbus-eth2/build/data/${NETWORK} \
--trusted-node-url=${TRUSTED_NODE_URL} \
--backfill=false
restart: "no"
# One-shot: downloads a pre-synced snapshot from snapshots.reth.rs into the
# Reth data dir. Turns a multi-day from-scratch sync into a ~hour download.
# Skipped if the data dir is already initialised — re-runs are no-ops.
# Privacy note: snapshots.reth.rs sees this download (operator existence).
# Subsequent eth_call traffic stays local.
reth-snapshot-init:
image: ghcr.io/paradigmxyz/reth:latest
depends_on:
init:
condition: service_completed_successfully
volumes:
- reth-data:/data
entrypoint:
- sh
- -c
- |
if [ -f /data/.snapshot-done ] || [ -d /data/db ]; then
echo "Reth data already initialised — skipping snapshot download";
exit 0;
fi;
echo "Downloading Reth ${NETWORK} --minimal snapshot...";
reth download --datadir /data --chain ${NETWORK} --minimal && \
touch /data/.snapshot-done && \
echo "Snapshot download complete"
restart: "no"
reth:
image: ghcr.io/paradigmxyz/reth:latest
depends_on:
reth-snapshot-init:
condition: service_completed_successfully
volumes:
- reth-data:/data
- jwt:/jwt:ro
ports:
# JSON-RPC for smp-server. Bound to loopback — put Caddy in front for remote access.
- "127.0.0.1:8545:8545"
# p2p (Ethereum network). Open these on your firewall for sync.
- "30303:30303/tcp"
- "30303:30303/udp"
command: >
node
--datadir /data
--chain ${NETWORK}
--minimal
--authrpc.jwtsecret /jwt/jwt.hex
--authrpc.addr 0.0.0.0 --authrpc.port 8551
--http
--http.addr 0.0.0.0 --http.port 8545
--http.api eth,net
--rpc.gascap 50000000
--port 30303
--discovery.port 30303
restart: unless-stopped
nimbus:
image: statusim/nimbus-eth2:multiarch-latest
depends_on:
nimbus-checkpoint-sync:
condition: service_completed_successfully
volumes:
- nimbus-data:/home/user/nimbus-eth2/build/data
- jwt:/jwt:ro
ports:
- "9000:9000/tcp"
- "9000:9000/udp"
- "127.0.0.1:5052:5052"
command: >
--network=${NETWORK}
--data-dir=/home/user/nimbus-eth2/build/data/${NETWORK}
--el=http://reth:8551
--jwt-secret=/jwt/jwt.hex
--non-interactive
--rest --rest-address=0.0.0.0 --rest-port=5052
--nat=${NAT:-any}
restart: unless-stopped
# SNRC REST resolver. Talks to reth on the compose-internal network,
# exposes /resolve and /health on 127.0.0.1:8000 by default. The
# smp-server points its [NAMES] resolver_endpoint at this URL.
# To change the host port, edit the LEFT side of the port mapping below.
resolver:
build:
context: ./service
dockerfile: Dockerfile
depends_on:
# reth's `service_started` is sufficient — the resolver tolerates
# eth_call failures gracefully (returns 502 with the error body), so
# starting before reth has finished snapshot replay just yields a few
# 502s until the chain is queryable. The upstream reth image doesn't
# ship a HEALTHCHECK, so we can't gate on healthy.
reth:
condition: service_started
environment:
SNRC_RPC: http://reth:8545
SNRC_BIND: 0.0.0.0
# Registry addresses cascade through the script's own defaults
# (mainnet `.testing`; `.simplex` unconfigured). Set explicitly here
# only if you're deploying against a different network or contract.
# SNRC_REGISTRY_TESTING: 0x...
# SNRC_REGISTRY_SIMPLEX: 0x...
ports:
- "127.0.0.1:8000:8000"
restart: unless-stopped
volumes:
reth-data:
nimbus-data:
jwt:
+48
View File
@@ -0,0 +1,48 @@
# syntax=docker/dockerfile:1.7
# ---------- builder ----------
# Use the official uv image (Astral) on top of a slim Python base.
# uv resolves and installs the lockfile-free pyproject.toml in seconds and
# produces a portable .venv we can copy into the runtime stage.
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS builder
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_NO_PROGRESS=1
WORKDIR /app
# Install deps first (separate layer) — script edits won't bust this cache.
COPY pyproject.toml ./
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --no-dev --no-install-project
# Script is added after the dep layer for cache friendliness.
COPY snrc-resolve.py ./
# ---------- runtime ----------
# Slim runtime — only the venv + script. No uv, no apt.
FROM python:3.13-slim AS runtime
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PATH="/app/.venv/bin:$PATH"
# Non-root user (matches resolver privacy posture: it has no need for root).
RUN groupadd --system --gid 10001 snrc && \
useradd --system --uid 10001 --gid snrc --no-create-home --shell /usr/sbin/nologin snrc
WORKDIR /app
COPY --from=builder --chown=snrc:snrc /app /app
USER snrc:snrc
EXPOSE 8000
# Liveness check hits the script's own /health route. ThreadingHTTPServer is
# fast enough that 3s is generous for a localhost probe; restart if it stops
# responding entirely.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD ["python", "-c", "import urllib.request, sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3).status == 200 else 1)"]
ENTRYPOINT ["python", "snrc-resolve.py"]
+13
View File
@@ -0,0 +1,13 @@
[project]
name = "snrc-resolve"
version = "0.1.0"
description = "SimpleX Namespace (SNRC) resolver — REST API over ENS-shaped Ethereum registries"
readme = "README.md"
requires-python = ">=3.11"
license = "AGPL-3.0-only"
dependencies = [
"eth-hash[pycryptodome]>=0.7",
]
[tool.uv]
package = false
+517
View File
@@ -0,0 +1,517 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "eth-hash[pycryptodome]>=0.7",
# ]
# ///
"""SimpleX Namespace (SNRC) resolver — REST API.
Resolves names like `alice.testing` / `bob.simplex` against the SNRC
deployment on Ethereum mainnet (or any compatible ENS-shaped registry)
and returns a flat JSON document with these fields:
name, nickname, website, location,
simplexContact, simplexChannel, -- list[str], primary first
eth, btc, xmr, dot,
owner, resolver
`simplexContact` and `simplexChannel` are arrays so a name can advertise
multiple SMP servers for redundancy. Clients SHOULD try the URLs in the
order returned. The on-chain text record stores them as a single
`LINK_SEPARATOR` (`;`)-joined string; this resolver splits and trims into a list.
All keys are valid Haskell record-field identifiers (lowercase initial,
no dots), so consumers can derive aeson FromJSON instances directly
without a key-rewriting layer.
Usage:
./snrc-resolve.py # serve on :8000
curl -s http://127.0.0.1:8000/resolve/foobar.testing | jq .
curl -s http://127.0.0.1:8000/health
Environment:
SNRC_RPC JSON-RPC endpoint (default: http://127.0.0.1:8545)
SNRC_REGISTRY_TESTING ENSRegistry for the .testing deployment
(default: mainnet,
0x58fc46996d975c57883564648bda5206d1a0102b)
SNRC_REGISTRY_SIMPLEX ENSRegistry for the .simplex deployment
(default: empty TLD not yet deployed)
SNRC_PORT Listen port (default: 8000)
SNRC_BIND Bind address (default: 0.0.0.0)
Each TLD is a separate SNRC deployment with its own ENSRegistry; the
resolver dispatches by the queried name's rightmost label.
Dependencies are declared inline (PEP 723) at the top of this file. Run with:
uv run snrc-resolve.py # uv resolves & caches deps; one-line setup
python snrc-resolve.py # if eth-hash[pycryptodome] is already installed
Addresses are returned in each chain's canonical presentation:
eth EIP-55 mixed-case checksummed hex (e.g. 0xEa65A01572)
btc bech32(m) for segwit/taproot, base58check for P2PKH/P2SH
(e.g. bc1q / 1A1zP1)
dot SS58 with Polkadot network prefix 0 (e.g. 15oF4u)
xmr Monero base58 (e.g. 4Aux5y)
Unrecognised payloads fall back to `0x`-prefixed raw hex.
"""
import hashlib
import json
import os
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import unquote, urlparse
from urllib.request import Request, urlopen
from eth_hash.auto import keccak
RPC = os.environ.get("SNRC_RPC", "http://127.0.0.1:8545")
BIND = os.environ.get("SNRC_BIND", "0.0.0.0")
PORT = int(os.environ.get("SNRC_PORT", "8000"))
# Each TLD is its own SNRC deployment with its own ENSRegistry. Dispatch
# happens on the rightmost label of the queried name. Empty / unset means
# "not deployed" — requests for that TLD return 400 with a clear error.
# `... or "..."` makes the script's defaults the single source of truth:
# unset AND empty-string both fall through to the literal. docker-compose
# can therefore pass `SNRC_REGISTRY_TESTING=${SNRC_REGISTRY_TESTING:-}`
# without duplicating the registry address.
REGISTRIES = {
"testing": os.environ.get("SNRC_REGISTRY_TESTING", "")
or "0x58fc46996d975c57883564648bda5206d1a0102b", # mainnet .testing
"simplex": os.environ.get("SNRC_REGISTRY_SIMPLEX", ""), # not deployed yet
}
# SLIP-44 coin types (https://github.com/satoshilabs/slips/blob/master/slip-0044.md)
COIN_ETH = 60
COIN_BTC = 0
COIN_XMR = 128
COIN_DOT = 354
ZERO_ADDR = "0x0000000000000000000000000000000000000000"
# ---------- RPC + ABI helpers (mirrors ens-lookup.py shape) ----------
def rpc(method, params):
body = json.dumps(
{"jsonrpc": "2.0", "method": method, "params": params, "id": 1}
).encode()
# Set a non-default User-Agent; Cloudflare-fronted public RPCs (drpc,
# publicnode, etc.) reject `Python-urllib/3.x` with 403.
req = Request(
RPC,
data=body,
headers={
"Content-Type": "application/json",
"User-Agent": "snrc-resolve/1.0",
},
)
res = json.loads(urlopen(req, timeout=15).read())
if "error" in res:
raise RuntimeError(res["error"])
return res["result"]
def namehash(name: str) -> bytes:
node = b"\x00" * 32
if name:
for label in reversed(name.split(".")):
node = keccak(node + keccak(label.encode()))
return node
def selector(signature: str) -> str:
return "0x" + keccak(signature.encode())[:4].hex()
def eth_call(to: str, data: str) -> str:
return rpc("eth_call", [{"to": to, "data": data}, "latest"])
def decode_address(hex_data: str) -> str:
return "0x" + hex_data[-40:]
def decode_bytes(hex_data: str) -> bytes:
raw = bytes.fromhex(hex_data[2:] if hex_data.startswith("0x") else hex_data)
if len(raw) < 64:
return b""
length = int.from_bytes(raw[32:64], "big")
return raw[64:64 + length]
def encode_text_call(node: bytes, key: str) -> str:
sel = selector("text(bytes32,string)")
head = node.hex() + (0x40).to_bytes(32, "big").hex()
key_bytes = key.encode()
body = len(key_bytes).to_bytes(32, "big").hex() + key_bytes.hex()
body += "00" * ((-len(key_bytes)) % 32)
return sel + head + body
def text(resolver: str, node: bytes, key: str) -> str:
raw = decode_bytes(eth_call(resolver, encode_text_call(node, key)))
return raw.decode("utf-8", errors="replace") if raw else ""
def encode_addr_multicoin_call(node: bytes, coin_type: int) -> str:
"""ENSIP-9 addr(bytes32 node, uint256 coinType) — both static, no offsets."""
return (
selector("addr(bytes32,uint256)")
+ node.hex()
+ coin_type.to_bytes(32, "big").hex()
)
def addr_multicoin(resolver: str, node: bytes, coin_type: int):
"""Read ENSIP-9 raw bytes for `coinType`, then encode to that chain's
canonical presentation form. Falls back to `0x`-prefixed hex if the
payload doesn't match any recognised on-chain shape. Returns None when
the record is unset."""
try:
raw = decode_bytes(eth_call(resolver, encode_addr_multicoin_call(node, coin_type)))
except RuntimeError:
return None
if not raw:
return None
# An all-zero payload is the ENS convention for "unset" — many tools
# write 20 zero bytes for coinType=60 instead of clearing the slot.
# Treat it as null so the response doesn't surface a zero address.
if raw == b"\x00" * len(raw):
return None
encoder = COIN_ENCODERS.get(coin_type)
if encoder is None:
return "0x" + raw.hex()
try:
return encoder(raw) or ("0x" + raw.hex())
except Exception:
return "0x" + raw.hex()
# ---------- Coin-specific address encoders ----------
# Each takes raw bytes as stored under ENSIP-9 and returns the canonical
# user-facing string for that chain (EIP-55 for ETH, bech32/base58check
# for BTC, SS58 for DOT, Monero-base58 for XMR). All stdlib + eth_hash.
B58_ALPHA = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
def _b58_encode(b: bytes) -> str:
n = int.from_bytes(b, "big")
out = ""
while n:
n, r = divmod(n, 58)
out = B58_ALPHA[r] + out
# leading zero bytes → leading '1's
pad = len(b) - len(b.lstrip(b"\x00"))
return "1" * pad + out
def _b58check_encode(payload: bytes) -> str:
"""Base58Check used by BTC legacy/P2SH: payload + dSHA256(payload)[:4]."""
chk = hashlib.sha256(hashlib.sha256(payload).digest()).digest()[:4]
return _b58_encode(payload + chk)
# ---- Bech32 / Bech32m (BIP-173 / BIP-350) ----
_BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
_BECH32_GEN = [0x3B6A57B2, 0x26508E6D, 0x1EA119FA, 0x3D4233DD, 0x2A1462B3]
def _bech32_polymod(values):
chk = 1
for v in values:
b = chk >> 25
chk = ((chk & 0x1FFFFFF) << 5) ^ v
for i in range(5):
if (b >> i) & 1:
chk ^= _BECH32_GEN[i]
return chk
def _bech32_hrp_expand(hrp):
return [ord(c) >> 5 for c in hrp] + [0] + [ord(c) & 31 for c in hrp]
def _bech32_create_checksum(hrp, data, spec):
const = 1 if spec == "bech32" else 0x2BC830A3 # bech32m
values = _bech32_hrp_expand(hrp) + data + [0] * 6
polymod = _bech32_polymod(values) ^ const
return [(polymod >> 5 * (5 - i)) & 31 for i in range(6)]
def _bech32_encode(hrp, data, spec):
combined = data + _bech32_create_checksum(hrp, data, spec)
return hrp + "1" + "".join(_BECH32_CHARSET[d] for d in combined)
def _convertbits(data, frombits, tobits, pad=True):
acc = 0
bits = 0
ret = []
maxv = (1 << tobits) - 1
max_acc = (1 << (frombits + tobits - 1)) - 1
for value in data:
if value < 0 or (value >> frombits):
return None
acc = ((acc << frombits) | value) & max_acc
bits += frombits
while bits >= tobits:
bits -= tobits
ret.append((acc >> bits) & maxv)
if pad and bits:
ret.append((acc << (tobits - bits)) & maxv)
elif not pad and (bits >= frombits or ((acc << (tobits - bits)) & maxv)):
return None
return ret
def _segwit_encode(hrp: str, witver: int, witprog: bytes) -> str:
spec = "bech32" if witver == 0 else "bech32m"
data = [witver] + _convertbits(list(witprog), 8, 5)
return _bech32_encode(hrp, data, spec)
# ---- BTC scriptPubKey → address ----
# ENSIP-9 stores the raw output script. Dispatch by length + opcode prefix.
def _btc_encode(raw: bytes) -> str | None:
hrp = "bc" # mainnet
if len(raw) == 25 and raw[:3] == b"\x76\xa9\x14" and raw[23:25] == b"\x88\xac":
return _b58check_encode(b"\x00" + raw[3:23]) # P2PKH
if len(raw) == 23 and raw[:2] == b"\xa9\x14" and raw[22:23] == b"\x87":
return _b58check_encode(b"\x05" + raw[2:22]) # P2SH
if len(raw) == 22 and raw[:2] == b"\x00\x14":
return _segwit_encode(hrp, 0, raw[2:22]) # P2WPKH
if len(raw) == 34 and raw[:2] == b"\x00\x20":
return _segwit_encode(hrp, 0, raw[2:34]) # P2WSH
if len(raw) == 34 and raw[:2] == b"\x51\x20":
return _segwit_encode(hrp, 1, raw[2:34]) # P2TR
return None
# ---- Polkadot SS58 ----
# Per SS58 spec: base58( prefix_byte + pubkey + blake2b-512("SS58PRE" + body)[:2] )
# Polkadot mainnet uses network prefix 0 (single byte); Kusama uses 2.
_SS58_PRE = b"SS58PRE"
def _ss58_encode(pubkey: bytes, network_prefix: int = 0) -> str:
if len(pubkey) != 32:
return None
body = bytes([network_prefix]) + pubkey
checksum = hashlib.blake2b(_SS58_PRE + body, digest_size=64).digest()[:2]
return _b58_encode(body + checksum)
def _dot_encode(raw: bytes) -> str | None:
return _ss58_encode(raw, network_prefix=0)
# ---- Monero base58 ----
# Monero base58 encodes in 8-byte blocks; each full block → 11 chars, partial
# block sizes per fixed table. Alphabet is identical to Bitcoin's.
_XMR_BLOCK_SIZES = [0, 2, 3, 5, 6, 7, 9, 10, 11]
def _xmr_encode(raw: bytes) -> str:
out = []
for i in range(0, len(raw), 8):
chunk = raw[i:i + 8]
n = int.from_bytes(chunk, "big")
width = 11 if len(chunk) == 8 else _XMR_BLOCK_SIZES[len(chunk)]
block = []
for _ in range(width):
n, r = divmod(n, 58)
block.append(B58_ALPHA[r])
out.append("".join(reversed(block)))
return "".join(out)
# ---- ETH EIP-55 mixed-case checksum ----
def _eth_encode(raw: bytes) -> str | None:
if len(raw) != 20:
return None
hex_addr = raw.hex()
hash_hex = keccak(hex_addr.encode()).hex()
return "0x" + "".join(
c.upper() if c.isalpha() and int(hash_hex[i], 16) >= 8 else c
for i, c in enumerate(hex_addr)
)
COIN_ENCODERS = {
COIN_ETH: _eth_encode,
COIN_BTC: _btc_encode,
COIN_XMR: _xmr_encode,
COIN_DOT: _dot_encode,
}
# ---------- Resolution logic ----------
# Text-record keys we read from the resolver. Surfaced under the response
# field names listed in the docstring above. `name` and `description` are
# common ENS fallbacks for a human-readable nickname.
TEXT_KEYS = [
"name",
"nickname",
"description",
"url",
"location",
"simplex.contact",
"simplex.channel",
]
# Separator that joins the SMP-server URL list inside a simplex.contact /
# simplex.channel text record. MUST match SIMPLEX_LINK_SEPARATOR in the dApp
# (ens-app-v3 src/constants/simplex.ts) — the two sides decode the same record.
LINK_SEPARATOR = ";"
def split_links(value: str) -> list:
"""Split a separator-joined text record into an ordered list of entries.
Trims whitespace around each element and drops empties so trailing
separators, doubled separators, and all-whitespace inputs all yield clean
output. Single-value records yield a 1-element list; empty inputs
yield `[]`. Used for `simplex.contact` / `simplex.channel`, which
store one-or-more SMP-server URLs as a single `LINK_SEPARATOR`-joined string.
"""
return [item.strip() for item in value.split(LINK_SEPARATOR) if item.strip()]
def resolve(name: str):
tld = name.rsplit(".", 1)[-1]
registry = REGISTRIES.get(tld)
if not registry:
configured = [k for k, v in REGISTRIES.items() if v]
return 400, {
"name": name,
"error": f"TLD '{tld}' is not configured on this resolver",
"configured_tlds": configured,
}
node = namehash(name)
node_hex = node.hex()
resolver_raw = eth_call(registry, selector("resolver(bytes32)") + node_hex)
resolver_addr = decode_address(resolver_raw)
if resolver_addr == ZERO_ADDR:
return 404, {"name": name, "error": "no resolver set for this name"}
owner_raw = eth_call(registry, selector("owner(bytes32)") + node_hex)
owner = decode_address(owner_raw)
texts = {}
for k in TEXT_KEYS:
try:
v = text(resolver_addr, node, k)
except RuntimeError:
v = ""
if v:
texts[k] = v
# The user-facing "nickname" prefers an explicit `nickname` record,
# falls back to `name`, then `description` (ENSIP-5 convention).
nickname = texts.get("nickname") or texts.get("name") or texts.get("description") or ""
# Keys chosen to be valid Haskell record-field identifiers (lowercase
# initial, no dots) so consumers can derive aeson FromJSON instances
# without a key-rewriting layer. On-chain text-record names still
# use the ENSIP-5 dot convention (e.g. "simplex.contact") — only the
# resolver's JSON surface camelCases them.
return 200, {
"name": name,
"nickname": nickname,
"website": texts.get("url", ""),
"location": texts.get("location", ""),
"simplexContact": split_links(texts.get("simplex.contact", "")),
"simplexChannel": split_links(texts.get("simplex.channel", "")),
"eth": addr_multicoin(resolver_addr, node, COIN_ETH),
"btc": addr_multicoin(resolver_addr, node, COIN_BTC),
"xmr": addr_multicoin(resolver_addr, node, COIN_XMR),
"dot": addr_multicoin(resolver_addr, node, COIN_DOT),
"owner": owner,
"resolver": resolver_addr,
}
# ---------- HTTP layer ----------
class Handler(BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802 - http.server contract
path = urlparse(self.path).path
parts = [unquote(p) for p in path.split("/") if p]
if parts == ["health"]:
self._respond(
200,
{"ok": True, "rpc": RPC, "registries": REGISTRIES},
)
return
if len(parts) == 2 and parts[0] == "resolve":
name = parts[1].strip().lower()
if not name or "." not in name:
self._respond(
400,
{
"error": "expected fully-qualified name, e.g. /resolve/alice.testing",
"got": name,
},
)
return
try:
status, body = resolve(name)
except Exception as e: # surface upstream errors as 502
status, body = 502, {"name": name, "error": f"{type(e).__name__}: {e}"}
self._respond(status, body)
return
self._respond(
404,
{"error": "not found", "routes": ["/health", "/resolve/<name>"]},
)
def _respond(self, status: int, body: dict):
data = json.dumps(body, indent=2).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, fmt, *args):
# Quiet the default per-request access log; route to stderr in one line.
sys.stderr.write(f"{self.address_string()} - {fmt % args}\n")
def main():
server = ThreadingHTTPServer((BIND, PORT), Handler)
sys.stderr.write(
f"snrc-resolve listening on {BIND}:{PORT}\n"
f" RPC = {RPC}\n"
f" Registries:\n"
)
for tld, addr in REGISTRIES.items():
sys.stderr.write(f" .{tld:<8s} = {addr or '(not configured)'}\n")
sys.stderr.write(" GET /resolve/<name> GET /health\n")
try:
server.serve_forever()
except KeyboardInterrupt:
sys.stderr.write("\nshutting down\n")
server.server_close()
if __name__ == "__main__":
main()
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""Unit tests for snrc-resolve helpers.
Run with `python3 -m unittest scripts/resolver/service/test_snrc_resolve.py`.
"""
import importlib.util
import os
import unittest
# snrc-resolve.py has a hyphen, so import it via importlib instead of `import`.
_HERE = os.path.dirname(os.path.abspath(__file__))
_SPEC = importlib.util.spec_from_file_location(
"snrc_resolve", os.path.join(_HERE, "snrc-resolve.py")
)
snrc = importlib.util.module_from_spec(_SPEC)
_SPEC.loader.exec_module(snrc)
class SplitLinksTests(unittest.TestCase):
"""`split_links` decodes the multi-URL convention for simplex.contact /
simplex.channel text records. Reuses the same rule the dApp's
`parseSimplexUrls` uses (separator `;`), so the two sides round-trip
cleanly."""
def test_empty_string_yields_empty_list(self):
self.assertEqual(snrc.split_links(""), [])
def test_whitespace_only_yields_empty_list(self):
self.assertEqual(snrc.split_links(" "), [])
self.assertEqual(snrc.split_links(" ; ; "), [])
def test_single_url_yields_singleton_list(self):
self.assertEqual(
snrc.split_links("https://smp16.simplex.im/a#H1"),
["https://smp16.simplex.im/a#H1"],
)
def test_two_urls_split_on_separator(self):
self.assertEqual(
snrc.split_links(
"https://smp16.simplex.im/a#H1;https://smp19.simplex.im/a#H1"
),
[
"https://smp16.simplex.im/a#H1",
"https://smp19.simplex.im/a#H1",
],
)
def test_whitespace_around_separators_is_trimmed(self):
self.assertEqual(
snrc.split_links(
" https://smp16.simplex.im/a#H1 ;\thttps://smp19.simplex.im/a#H1 "
),
[
"https://smp16.simplex.im/a#H1",
"https://smp19.simplex.im/a#H1",
],
)
def test_trailing_separator_does_not_produce_empty_entry(self):
self.assertEqual(
snrc.split_links("https://smp16.simplex.im/a#H1;"),
["https://smp16.simplex.im/a#H1"],
)
def test_doubled_separator_does_not_produce_empty_entry(self):
self.assertEqual(
snrc.split_links(
"https://smp16.simplex.im/a#H1;;https://smp19.simplex.im/a#H1"
),
[
"https://smp16.simplex.im/a#H1",
"https://smp19.simplex.im/a#H1",
],
)
def test_order_is_preserved(self):
self.assertEqual(
snrc.split_links("c;a;b"),
["c", "a", "b"],
)
if __name__ == "__main__":
unittest.main()
+168
View File
@@ -0,0 +1,168 @@
#!/usr/bin/env python3
"""Resolve an ENS name via local Reth (the same shape SNRC will use).
Usage:
./ens-lookup.py # defaults to simplexchat.eth
./ens-lookup.py vitalik.eth
./ens-lookup.py corevo.eth
Requires: pip install --break-system-packages 'eth-hash[pycryptodome]'
"""
import base64
import json
import sys
from urllib.request import Request, urlopen
from eth_hash.auto import keccak
RPC = "http://127.0.0.1:8545"
# ENS Registry (current, post-2020 migration)
ENS_REGISTRY = "0x00000000000C2E074eC69A0dFb2997BA6C7d2e1e"
def rpc(method, params):
body = json.dumps({"jsonrpc": "2.0", "method": method, "params": params, "id": 1}).encode()
req = Request(RPC, data=body, headers={"Content-Type": "application/json"})
res = json.loads(urlopen(req, timeout=15).read())
if "error" in res:
raise RuntimeError(res["error"])
return res["result"]
def namehash(name: str) -> bytes:
"""ENS namehash — recursive keccak256 over reversed labels."""
node = b"\x00" * 32
if name:
for label in reversed(name.split(".")):
node = keccak(node + keccak(label.encode()))
return node
def selector(signature: str) -> str:
return "0x" + keccak(signature.encode())[:4].hex()
def eth_call(to: str, data: str) -> str:
return rpc("eth_call", [{"to": to, "data": data}, "latest"])
def decode_address(hex_data: str) -> str:
return "0x" + hex_data[-40:]
def decode_bytes(hex_data: str) -> bytes:
raw = bytes.fromhex(hex_data[2:] if hex_data.startswith("0x") else hex_data)
if len(raw) < 64:
return b""
length = int.from_bytes(raw[32:64], "big")
return raw[64:64 + length]
def encode_text_call(node: bytes, key: str) -> str:
"""ABI-encode text(bytes32 node, string key). String arg is dynamic:
offset (=0x40) + length + right-padded data."""
sel = selector("text(bytes32,string)")
head = node.hex() + (0x40).to_bytes(32, "big").hex()
key_bytes = key.encode()
body = len(key_bytes).to_bytes(32, "big").hex() + key_bytes.hex()
# right-pad to 32-byte boundary
pad = (-len(key_bytes)) % 32
body += "00" * pad
return sel + head + body
def text(resolver: str, node: bytes, key: str) -> str:
raw = decode_bytes(eth_call(resolver, encode_text_call(node, key)))
return raw.decode("utf-8", errors="replace") if raw else ""
# Common ENS text keys (ENSIP-5). Resolvers may return empty for any of these.
TEXT_KEYS = [
"url",
"avatar",
"description",
"email",
"notice",
"keywords",
"com.twitter",
"com.github",
"com.discord",
"org.telegram",
"io.keybase",
"xyz.farcaster",
]
def decode_contenthash(raw: bytes) -> str:
"""ENS contenthash → human-readable URI (best-effort)."""
if not raw:
return "(empty)"
# Multicodec prefixes:
# 0xe301 = ipfs-ns + dag-pb (CIDv0/v1)
# 0xe501 = ipns-ns
# 0xe40101701b... = swarm
if raw[:2] == b"\xe3\x01":
cid_bytes = raw[2:]
# Base32 lowercase + 'b' prefix per CIDv1 spec
b32 = base64.b32encode(cid_bytes).decode().lower().rstrip("=")
return f"ipfs://b{b32}"
if raw[:2] == b"\xe5\x01":
cid_bytes = raw[2:]
b32 = base64.b32encode(cid_bytes).decode().lower().rstrip("=")
return f"ipns://b{b32}"
return "0x" + raw.hex()
def main():
name = sys.argv[1] if len(sys.argv) > 1 else "simplexchat.eth"
print(f" name: {name}")
node = namehash(name)
print(f" namehash: 0x{node.hex()}")
# 1. Ask the registry which resolver is responsible for this name
resolver_data = selector("resolver(bytes32)") + node.hex()
resolver_raw = eth_call(ENS_REGISTRY, resolver_data)
resolver = decode_address(resolver_raw)
print(f" resolver: {resolver}")
if resolver == "0x0000000000000000000000000000000000000000":
print(" → no resolver set for this name")
return
node_hex = node.hex()
# 2. Ask the resolver for the address
try:
addr = decode_address(eth_call(resolver, selector("addr(bytes32)") + node_hex))
print(f" address: {addr}")
except Exception as e:
print(f" address: (error: {e})")
# 3. Ask the resolver for the content hash (IPFS pointer)
try:
ch = decode_bytes(eth_call(resolver, selector("contenthash(bytes32)") + node_hex))
print(f" contenthash: {decode_contenthash(ch)}")
except Exception as e:
print(f" contenthash: (not supported: {e})")
# 4. Owner from the registry
try:
owner = decode_address(eth_call(ENS_REGISTRY, selector("owner(bytes32)") + node_hex))
print(f" owner: {owner}")
except Exception as e:
print(f" owner: (error: {e})")
# 5. Text records (EIP-634). Print only the non-empty ones.
print(" text records:")
for key in TEXT_KEYS:
try:
v = text(resolver, node, key)
if v:
print(f" {key:<16s} {v}")
except Exception as e:
print(f" {key:<16s} (error: {e})")
if __name__ == "__main__":
main()
+246
View File
@@ -0,0 +1,246 @@
#!/usr/bin/env python3
"""Sync progress for the Reth + Nimbus stack.
Usage:
./progress.py # continuous (Ctrl-C to exit, auto-exits when synced)
./progress.py --once # single snapshot
Requires Nimbus REST port exposed at 127.0.0.1:5052 (add --rest flag in compose).
"""
import json
import sys
import time
from collections import deque
from datetime import timedelta
from urllib.error import URLError
from urllib.request import Request, urlopen
RETH = "http://127.0.0.1:8545"
NIMBUS = "http://127.0.0.1:5052"
INTERVAL = 5
WINDOW = 60
BAR_W = 40
# ANSI helpers
def c(s, code): return f"\033[{code}m{s}\033[0m"
GREEN, YELLOW, RED, DIM, BOLD = "32", "33", "31", "2;37", "1"
def rpc(method):
body = json.dumps({"jsonrpc": "2.0", "method": method, "params": [], "id": 1}).encode()
req = Request(RETH, data=body, headers={"Content-Type": "application/json"})
return json.loads(urlopen(req, timeout=5).read())["result"]
def get_reth():
try:
r = rpc("eth_syncing")
try:
peers = int(rpc("net_peerCount"), 16)
except Exception:
peers = -1 # net namespace not exposed
if r is False:
head = int(rpc("eth_blockNumber"), 16)
return {"state": "synced", "current": head, "target": head, "peers": peers,
"stage": None, "stages": {}, "err": None}
current = int(r["currentBlock"], 16)
highest = int(r["highestBlock"], 16)
# Build stage map (name -> block).
stages = {s["name"]: int(s["block"], 16) for s in r.get("stages", [])}
active_stages = {k: v for k, v in stages.items() if v > 0}
# Headers download phase: nothing has progressed yet.
if current == 0 and highest == 0 and not active_stages:
return {"state": "headers", "current": 0, "target": 0, "peers": peers,
"stage": "Headers", "stages": stages, "err": None}
# Derive progress from the stages pipeline.
# Bottleneck (rate-limiting stage) = stage with lowest non-zero block.
# Target = leading stage block (typically Headers = chain tip).
# Reth's top-level currentBlock/highestBlock are unreliable during initial
# sync (often 0 until execution stage runs), so prefer stages-derived values.
if active_stages:
bottleneck = min(active_stages, key=active_stages.get)
stage_current = active_stages[bottleneck]
stage_target = max(stages.values()) if stages else 0
# Trust the stages-derived values if highest is unset or stages tip is higher.
if highest <= 0 or stage_target > highest:
current = stage_current
highest = stage_target
elif current <= 0:
current = stage_current
else:
bottleneck = None
return {"state": "syncing", "current": current, "target": highest,
"peers": peers, "stage": bottleneck, "stages": stages, "err": None}
except URLError as e:
return {"state": "down", "current": 0, "target": 0, "peers": 0,
"stage": None, "stages": {}, "err": str(e.reason)}
except Exception as e:
return {"state": "error", "current": 0, "target": 0, "peers": 0,
"stage": None, "stages": {}, "err": str(e)}
def get_nimbus():
try:
d = json.loads(urlopen(f"{NIMBUS}/eth/v1/node/syncing", timeout=5).read())["data"]
peers_d = json.loads(urlopen(f"{NIMBUS}/eth/v1/node/peer_count", timeout=5).read())["data"]
head = int(d["head_slot"])
dist = int(d["sync_distance"])
peers = int(peers_d.get("connected", "0"))
return {"state": "synced" if not d["is_syncing"] else "syncing",
"current": head, "target": head + dist, "peers": peers,
"optimistic": bool(d.get("is_optimistic", False)),
"el_offline": bool(d.get("el_offline", False)),
"err": None}
except URLError as e:
return {"state": "down", "current": 0, "target": 0, "peers": 0,
"optimistic": False, "el_offline": False, "err": str(e.reason)}
except Exception as e:
return {"state": "error", "current": 0, "target": 0, "peers": 0,
"optimistic": False, "el_offline": False, "err": str(e)}
def format_num(n): return f"{n:,}"
def format_eta(seconds):
if seconds is None: return "?"
if seconds < 0: return "?"
if seconds < 60: return f"{int(seconds)}s"
if seconds < 3600:
return f"{int(seconds // 60)}m {int(seconds % 60)}s"
if seconds < 86400:
return f"{int(seconds // 3600)}h {int((seconds % 3600) // 60)}m"
return f"{int(seconds // 86400)}d {int((seconds % 86400) // 3600)}h"
def rate_per_sec(history):
if len(history) < 2: return None
t0, c0 = history[0]
t1, c1 = history[-1]
if t1 <= t0: return None
return (c1 - c0) / (t1 - t0)
def eta_seconds(history, target):
r = rate_per_sec(history)
if r is None or r <= 0: return None
remaining = target - history[-1][1]
if remaining <= 0: return 0
return remaining / r
def progress_bar(pct):
pct = max(0.0, min(100.0, pct))
filled = int(pct / 100 * BAR_W)
return c("" * filled, GREEN) + c("" * (BAR_W - filled), DIM)
def peers_label(peers):
if peers < 0:
return c("· peers unknown (enable net namespace)", DIM)
return c(f"· {peers} peers", DIM)
def stages_summary(stages):
"""One-line view: stages that have progressed, with their block numbers."""
if not stages:
return ""
advanced = [(n, b) for n, b in stages.items() if b > 0]
if not advanced:
return c(" stages: all 0 (headers downloading)", DIM)
advanced.sort(key=lambda kv: kv[1], reverse=True)
parts = [f"{n}={format_num(b)}" for n, b in advanced[:4]]
return c(" stages: " + ", ".join(parts), DIM)
def render_one(name, x, hist):
state = x["state"]
peers = x.get("peers", 0)
extras = []
if name == "Nimbus":
if x.get("optimistic"):
extras.append(c("(optimistic head — Reth not yet verifying)", YELLOW))
if x.get("el_offline"):
extras.append(c("⚠ EL OFFLINE", RED))
if state == "synced":
out = [f" {c(name, BOLD):<14s} {c('✓ synced', GREEN)} {c(format_num(x['current']), BOLD)} {peers_label(peers)}"]
elif state == "headers":
out = [
f" {c(name, BOLD):<14s} {c('⧗ headers', YELLOW)} {c('downloading initial chain', DIM)} {peers_label(peers)}",
f" {c('(per-block progress unavailable until headers validated — see docker logs)', DIM)}",
]
elif state == "syncing" and x["target"] <= 0:
out = [f" {c(name, BOLD):<14s} {c('⧗ syncing', YELLOW)} {c('waiting for fork-choice', DIM)} {peers_label(peers)}"]
elif state == "syncing":
pct = x["current"] / x["target"] * 100
r = rate_per_sec(hist)
eta = eta_seconds(hist, x["target"])
rate_s = f"{format_num(int(r))} /s" if r and r > 0 else c("stalled", RED)
eta_s = format_eta(eta) if eta is not None else "?"
stage = x.get("stage")
stage_s = c(f"[{stage}]", DIM) if stage else ""
out = [
f" {c(name, BOLD):<14s} {c('⧗ syncing', YELLOW)} {format_num(x['current'])} / {format_num(x['target'])} {stage_s} {peers_label(peers)}",
f" {progress_bar(pct)} {c(f'{pct:6.2f}%', BOLD)}",
f" {c(rate_s, DIM)} ETA {c(eta_s, BOLD)}",
]
else:
out = [
f" {c(name, BOLD):<14s} {c('' + state, RED)}",
f" {c(x.get('err') or '', DIM)}",
]
# Reth-only: stages summary
if name == "Reth" and x.get("stages"):
out.append(f" {stages_summary(x['stages'])}")
for e in extras:
out.append(f" {e}")
return out
def render(reth, nimbus, reth_hist, nim_hist):
print("\033[2J\033[H", end="")
width = 64
title = f"Reth + Nimbus sync"
ts = time.strftime("%H:%M:%S")
print()
print(f" {c(title, BOLD)} {c(ts, DIM)}")
print(f" {c('' * width, DIM)}")
print()
for line in render_one("Reth", reth, reth_hist):
print(line)
print()
for line in render_one("Nimbus", nimbus, nim_hist):
print(line)
print()
win_s = (len(reth_hist) - 1) * INTERVAL if len(reth_hist) > 1 else 0
print(f" {c(f'window {win_s}s · refresh {INTERVAL}s · Ctrl-C to exit', DIM)}")
print()
def main():
once = "--once" in sys.argv
reth_hist = deque(maxlen=WINDOW)
nim_hist = deque(maxlen=WINDOW)
try:
while True:
r = get_reth()
n = get_nimbus()
now = time.time()
if r["target"] > 0 or r["state"] == "syncing":
reth_hist.append((now, r["current"]))
if n["target"] > 0 or n["state"] == "syncing":
nim_hist.append((now, n["current"]))
render(r, n, reth_hist, nim_hist)
if once:
break
if r["state"] == "synced" and n["state"] == "synced":
print(f" {c('✓ all synced.', GREEN)}\n")
break
time.sleep(INTERVAL)
except KeyboardInterrupt:
print()
if __name__ == "__main__":
main()
@@ -22,7 +22,7 @@ smp-server --version
# Initialize server
ip_address=$(curl ifconfig.me)
smp-server init -l --ip $ip_address
smp-server init -l --disable-web --ip $ip_address
# Server fingerprint
fingerprint=$(cat /etc/opt/simplex/fingerprint)

Some files were not shown because too many files have changed in this diff Show More