From a7c6dde39f9e145debe69b121f811c4003857ec6 Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sat, 14 Mar 2026 09:07:26 +0000 Subject: [PATCH] router diagrams --- docs/ROUTERS.md | 37 +--- spec/diagrams/ntf-router.svg | 208 ++++++++++++++++++ spec/diagrams/smp-router.svg | 191 ++++++++++++++++ spec/diagrams/xftp-router.svg | 133 +++++++++++ spec/modules/Simplex/FileTransfer/Server.md | 2 + .../Simplex/Messaging/Notifications/Server.md | 2 + spec/modules/Simplex/Messaging/Server.md | 2 + spec/routers.md | 164 ++++++++++++++ 8 files changed, 707 insertions(+), 32 deletions(-) create mode 100644 spec/diagrams/ntf-router.svg create mode 100644 spec/diagrams/smp-router.svg create mode 100644 spec/diagrams/xftp-router.svg create mode 100644 spec/routers.md diff --git a/docs/ROUTERS.md b/docs/ROUTERS.md index 7ebc0f9ee..29d518d65 100644 --- a/docs/ROUTERS.md +++ b/docs/ROUTERS.md @@ -2,11 +2,11 @@ SimpleX routers are the network infrastructure of the [SimpleX Network](../protocol/overview-tjr.md). They accept, buffer, and deliver data packets between endpoints. Each router operates independently and can be run by any party on standard computing hardware. -This document covers deployment and advanced configuration. For an overview of the router architecture and trust model, see the [SimpleX Network overview](../protocol/overview-tjr.md). +This document covers deployment and advanced configuration. For an overview of the router architecture and trust model, see the [SimpleX Network overview](../protocol/overview-tjr.md). For internal architecture diagrams (thread topology, command processing flows), see [`spec/routers.md`](../spec/routers.md). ## SMP Router -The SMP router provides messaging queues — unidirectional, ordered sequences of fixed-size packets (16,384 bytes each). It implements the [SimpleX Messaging Protocol](../protocol/simplex-messaging.md). **Module spec**: [`spec/modules/Simplex/Messaging/Server.md`](../spec/modules/Simplex/Messaging/Server.md). +The SMP router provides messaging queues — unidirectional, ordered sequences of fixed-size packets (16,384 bytes each). It implements the [SimpleX Messaging Protocol](../protocol/simplex-messaging.md). For architecture and module specs, see [SMP Router](../spec/routers.md#smp-router). ### Advanced configuration @@ -35,13 +35,13 @@ echo 'PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"' >> ~/.zprofile ## XFTP Router -The XFTP router accepts and delivers data packets over HTTP/2 — individually addressed blocks in fixed sizes (64KB, 256KB, 1MB, 4MB). It implements the [XFTP protocol](../protocol/xftp.md). Data packets are used for larger payload delivery (files, media) where SMP queue packet sizes would be inefficient. The use of HTTP/2 simplifies browser integration. **Module spec**: [`spec/modules/Simplex/FileTransfer/Server.md`](../spec/modules/Simplex/FileTransfer/Server.md). +The XFTP router accepts and delivers data packets over HTTP/2 — individually addressed blocks in fixed sizes (64KB, 256KB, 1MB, 4MB). It implements the [XFTP protocol](../protocol/xftp.md). Data packets are used for larger payload delivery (files, media) where SMP queue packet sizes would be inefficient. The use of HTTP/2 simplifies browser integration. For architecture and module specs, see [XFTP Router](../spec/routers.md#xftp-router). Initialize with `xftp-server init` and configure storage quota in `xftp-server.ini`. ## NTF Router -The NTF router bridges SimpleX Network to platform push notification services (APNS). It implements the [Push Notifications protocol](../protocol/push-notifications.md). Mobile clients register push tokens with the NTF router, which subscribes to their SMP queues and sends push notifications when messages arrive. The push notification contains only a notification ID, not message content. **Module spec**: [`spec/modules/Simplex/Messaging/Notifications/Server.md`](../spec/modules/Simplex/Messaging/Notifications/Server.md). +The NTF router bridges SimpleX Network to platform push notification services (APNS). It implements the [Push Notifications protocol](../protocol/push-notifications.md). Mobile clients register push tokens with the NTF router, which subscribes to their SMP queues and sends push notifications when messages arrive. The push notification contains only a notification ID, not message content. For architecture and module specs, see [NTF Router](../spec/routers.md#ntf-router). Initialize with `ntf-server init` and configure APNS credentials in `ntf-server.ini`. @@ -184,7 +184,7 @@ smp-server init [-l] -n ## Monitoring -SMP and XFTP routers expose Prometheus metrics via a control port. The control port also supports commands for runtime inspection (queue counts, client counts, statistics). See [SMP Server Prometheus](../spec/modules/Simplex/Messaging/Server/Prometheus.md), [SMP Server Control](../spec/modules/Simplex/Messaging/Server/Control.md), and [NTF Server Control](../spec/modules/Simplex/Messaging/Notifications/Server/Control.md) module specs for available metrics and control commands. +SMP and XFTP routers expose Prometheus metrics via a control port. The control port also supports commands for runtime inspection (queue counts, client counts, statistics). See module specs linked from each router section in [`spec/routers.md`](../spec/routers.md) (Control, Prometheus, Stats). ## Protocol references @@ -192,30 +192,3 @@ SMP and XFTP routers expose Prometheus metrics via a control port. The control p - [XFTP Protocol](../protocol/xftp.md) — data packet protocol - [Push Notifications Protocol](../protocol/push-notifications.md) — NTF protocol - [SimpleX Network overview](../protocol/overview-tjr.md) — architecture and trust model - -## Module specs - -### SMP Router -- [Server](../spec/modules/Simplex/Messaging/Server.md) — main server module, client handling, message routing -- [Server Main](../spec/modules/Simplex/Messaging/Server/Main.md) — server startup, initialization -- [QueueStore](../spec/modules/Simplex/Messaging/Server/QueueStore.md) — queue persistence abstraction -- [QueueStore Postgres](../spec/modules/Simplex/Messaging/Server/QueueStore/Postgres.md) — PostgreSQL queue store -- [MsgStore](../spec/modules/Simplex/Messaging/Server/MsgStore.md) — message storage abstraction -- [StoreLog](../spec/modules/Simplex/Messaging/Server/StoreLog.md) — append-only store log for queue persistence -- [Server Control](../spec/modules/Simplex/Messaging/Server/Control.md) — control port commands -- [Server Prometheus](../spec/modules/Simplex/Messaging/Server/Prometheus.md) — metrics export -- [Server Stats](../spec/modules/Simplex/Messaging/Server/Stats.md) — statistics collection - -### XFTP Router -- [Server](../spec/modules/Simplex/FileTransfer/Server.md) — main server module, data packet handling -- [Server Main](../spec/modules/Simplex/FileTransfer/Server/Main.md) — server startup -- [Server Store](../spec/modules/Simplex/FileTransfer/Server/Store.md) — data packet storage -- [Server StoreLog](../spec/modules/Simplex/FileTransfer/Server/StoreLog.md) — store log for packet persistence -- [Server Stats](../spec/modules/Simplex/FileTransfer/Server/Stats.md) — statistics - -### NTF Router -- [Server](../spec/modules/Simplex/Messaging/Notifications/Server.md) — main server module -- [Server Main](../spec/modules/Simplex/Messaging/Notifications/Server/Main.md) — server startup -- [Server Store Postgres](../spec/modules/Simplex/Messaging/Notifications/Server/Store/Postgres.md) — PostgreSQL store for tokens and subscriptions -- [APNS Push](../spec/modules/Simplex/Messaging/Notifications/Server/Push/APNS.md) — Apple push notification delivery -- [Server Control](../spec/modules/Simplex/Messaging/Notifications/Server/Control.md) — control port commands diff --git a/spec/diagrams/ntf-router.svg b/spec/diagrams/ntf-router.svg new file mode 100644 index 000000000..fb35fe804 --- /dev/null +++ b/spec/diagrams/ntf-router.svg @@ -0,0 +1,208 @@ + + + + + + + + + + + + NTF Router -- Component Topology + + + + per client (raceAny_) + + + net + + + + + receive + + + + rcvQ + + + + client + + + + sndQ + + + + send + + + net + + + TNEW, TVFY, TRPL, TDEL + SNEW, SCHK, SDEL + + + + store + + + + SMP Client Agent (connects to SMP routers) + + + SMP routers + + + + + SMPClientAgent + + + + msgQ + + + + agentQ + + + + ntfSubscriber/receiveSMP + + + + receiveAgent + + + race_ + + + + pushQ + + + + store + + + + runSMPSubscriber + (one per SMP router) + + + subscriberSubQ + + + + tokens / subscriptions / tokenLastNtfs + (in-memory TMap + PostgreSQL) + + + + push delivery pipeline + + + + pushQ + + + + ntfPush + + + + + APNS provider + + + + periodicNtfsThread + + + + pushQ + + + + reads + + + + optional + + + logServerStats + + + prometheus + + + controlPort + + + resubscribe + + + + + + per-client thread + + + singleton thread + + + storage + + + external connection + + + Solid arrows: TBQueue connections. Dashed: store access. + + + diff --git a/spec/diagrams/smp-router.svg b/spec/diagrams/smp-router.svg new file mode 100644 index 000000000..796bbf80f --- /dev/null +++ b/spec/diagrams/smp-router.svg @@ -0,0 +1,191 @@ + + + + + + + + + + + + SMP Router -- Component Topology + + + + per client connection (raceAny_ -- any thread exit tears down connection) + + + + receive + + + + rcvQ + + + + client + + + + sndQ + + + + send + + + + msgQ + + + + sendMsg + + + net + + + + + + net + + + + + + + + + + QueueStore + (STM or Postgres) + + + + MsgStore + (STM or Postgres) + + + + StoreLog (optional) + + + + + subQ + + + + singleton threads (one instance each, all in raceAny_) + + + + serverThread + (SMP subscriptions) + + + + serverThread + (NTF subscriptions) + + + + pendingEvents + + + + deliverNtfs + + + + sendPendingEvts + + + + expireMessages + + + + expireNtfs + + + + proxyAgent + + + + optional + + + + logServerStats + + + + prometheus + + + + controlPort + + + + + + per-client thread + + + singleton thread + + + storage + + + optional + + + Solid arrows: TBQueue connections. Dashed blue: subQ linking per-client to singleton threads. + + + diff --git a/spec/diagrams/xftp-router.svg b/spec/diagrams/xftp-router.svg new file mode 100644 index 000000000..bf60f000d --- /dev/null +++ b/spec/diagrams/xftp-router.svg @@ -0,0 +1,133 @@ + + + + + + + + + + + + XFTP Router -- Component Topology + + + + per request (inline HTTP/2 callback, no spawned threads) + + + net + + + + + HTTP/2 handler + + + + Handshake State (per session) + None -> Sent -> Accepted + + + + sessions + + + + + + + Command Processing (FNEW, FADD, FPUT, FGET, FACK, FDEL) + + + + + + + + + + FileStore + (TMap in STM) + + + + Disk Storage + filesPath / senderId / data + + + quota-managed via usedStorage TVar + + + + StoreLog (append-only) + + + + + + + net + + + + background threads (singleton, in raceAny_) + + + + expireFiles + + + + logServerStats + + + + prometheus + + + + controlPort + + + + + + request handler (no threads) + + + storage + + + per-session state + + + background thread + + diff --git a/spec/modules/Simplex/FileTransfer/Server.md b/spec/modules/Simplex/FileTransfer/Server.md index cb64adad2..b695fe908 100644 --- a/spec/modules/Simplex/FileTransfer/Server.md +++ b/spec/modules/Simplex/FileTransfer/Server.md @@ -16,6 +16,8 @@ The XFTP router runs several concurrent threads via `raceAny_`: | `savePrometheusMetrics` | Periodic Prometheus metrics dump | | `runCPServer` | Control port for admin commands | +See [spec/routers.md](../../routers.md) for component and sequence diagrams. + ## Non-obvious behavior ### 1. Three-state handshake with session caching diff --git a/spec/modules/Simplex/Messaging/Notifications/Server.md b/spec/modules/Simplex/Messaging/Notifications/Server.md index b87f64ce8..0f7ebc67d 100644 --- a/spec/modules/Simplex/Messaging/Notifications/Server.md +++ b/spec/modules/Simplex/Messaging/Notifications/Server.md @@ -18,6 +18,8 @@ The NTF router runs several concurrent threads via `raceAny_`: Each client connection spawns `receive`, `send`, and `client` threads via `raceAny_`. +See [spec/routers.md](../../../routers.md) for component and sequence diagrams. + ## Non-obvious behavior ### 1. Timing attack mitigation on entity lookup diff --git a/spec/modules/Simplex/Messaging/Server.md b/spec/modules/Simplex/Messaging/Server.md index 5cfdfa24a..7d991fbb7 100644 --- a/spec/modules/Simplex/Messaging/Server.md +++ b/spec/modules/Simplex/Messaging/Server.md @@ -10,6 +10,8 @@ The router runs as `raceAny_` over many threads — any thread exit stops the entire router process. The thread set includes: one `serverThread` per subscription type (SMP, NTF), a notification delivery thread, a pending events thread, a proxy agent receiver, a SIGINT handler, plus per-transport listener threads and optional expiration/stats/prometheus/control-port threads. `E.finally` ensures `stopServer` runs on any exit. +See [spec/routers.md](../../routers.md) for component and sequence diagrams. + ## serverThread — subscription lifecycle with split STM See comment on `serverThread`. It reads the subscription request from `subQ`, then looks up the client **outside** STM (via `getServerClient`), then enters an STM transaction (`updateSubscribers`) to compute which old subscriptions to end, then runs `endPreviousSubscriptions` in IO. If the client disconnects between lookup and transaction, `updateSubscribers` handles `Nothing` by still sending END/DELD to other subscribed clients. diff --git a/spec/routers.md b/spec/routers.md new file mode 100644 index 000000000..f146ca8af --- /dev/null +++ b/spec/routers.md @@ -0,0 +1,164 @@ +# Router Architecture + +SimpleX routers are the Layer 1 network infrastructure. This document shows their internal architecture: component topology and command processing flows. + +For deployment and configuration, see [docs/ROUTERS.md](../docs/ROUTERS.md). For protocol specifications, see [SMP](../protocol/simplex-messaging.md), [XFTP](../protocol/xftp.md), [Push Notifications](../protocol/push-notifications.md). + +--- + +## SMP Router + +**Module specs**: [Server](modules/Simplex/Messaging/Server.md) · [Main](modules/Simplex/Messaging/Server/Main.md) · [QueueStore](modules/Simplex/Messaging/Server/QueueStore.md) · [QueueStore Postgres](modules/Simplex/Messaging/Server/QueueStore/Postgres.md) · [MsgStore](modules/Simplex/Messaging/Server/MsgStore.md) · [StoreLog](modules/Simplex/Messaging/Server/StoreLog.md) · [Control](modules/Simplex/Messaging/Server/Control.md) · [Prometheus](modules/Simplex/Messaging/Server/Prometheus.md) · [Stats](modules/Simplex/Messaging/Server/Stats.md) + +### Component topology + +![SMP Router — Component Topology](diagrams/smp-router.svg) + +### Packet delivery flow + +```mermaid +sequenceDiagram + participant S as Sender + + box SMP Router + participant auth as Command
Authorization + participant QS as QueueStore + participant MS as MsgStore + participant del as Packet
Delivery + end + + participant R as Recipient + + S->>auth: SEND (queue ID + packet) + auth->>QS: verify sender key (constant-time) + auth->>MS: store packet + auth->>S: OK (via sndQ) + + auth->>del: tryDeliverMessage + + alt recipient has active SUB + del->>R: MSG (via recipient's sndQ) + R->>auth: ACK + auth->>MS: delete packet + else no active subscriber + Note over MS: packet waits in MsgStore + R->>auth: SUB (subscribe to queue) + auth->>MS: fetch pending packets + del->>R: MSG + end +``` + +### Proxy forwarding flow + +```mermaid +sequenceDiagram + participant C as Client + participant P as Proxy Router + participant D as Destination Router + + C->>P: PRXY (destination address) + P->>D: connect (if not already connected) + P->>C: PKEY (proxy session key) + + C->>P: PFWD (encrypted command for destination) + P->>D: RFWD (relay forwarded command) + D->>P: command result + P->>C: command result +``` + +--- + +## XFTP Router + +**Module specs**: [Server](modules/Simplex/FileTransfer/Server.md) · [Main](modules/Simplex/FileTransfer/Server/Main.md) · [Store](modules/Simplex/FileTransfer/Server/Store.md) · [StoreLog](modules/Simplex/FileTransfer/Server/StoreLog.md) · [Stats](modules/Simplex/FileTransfer/Server/Stats.md) · [Transport](modules/Simplex/FileTransfer/Transport.md) + +### Component topology + +![XFTP Router — Component Topology](diagrams/xftp-router.svg) + +### Data packet delivery flow + +```mermaid +sequenceDiagram + participant S as Sender + + box XFTP Router + participant HS as Handshake + participant CP as Command
Processing + participant FS as FileStore + participant D as Disk + end + + participant R as Recipient + + S->>HS: HELLO + HS->>S: server DH key + version + + S->>CP: FNEW (create data packet) + CP->>FS: create FileRec, reserve quota + CP->>S: sender ID + recipient IDs + + S->>CP: FPUT (send encrypted data) + CP->>D: write to disk + CP->>FS: commit filePath + CP->>S: OK + + R->>HS: HELLO + HS->>R: server DH key + version + + R->>CP: FGET (recipient DH key) + CP->>CP: DH key agreement + CP->>D: read file + CP->>R: encrypted data stream + + R->>CP: FACK + CP->>FS: delete recipient entry +``` + +--- + +## NTF Router + +**Module specs**: [Server](modules/Simplex/Messaging/Notifications/Server.md) · [Main](modules/Simplex/Messaging/Notifications/Server/Main.md) · [Store Postgres](modules/Simplex/Messaging/Notifications/Server/Store/Postgres.md) · [APNS](modules/Simplex/Messaging/Notifications/Server/Push/APNS.md) · [Control](modules/Simplex/Messaging/Notifications/Server/Control.md) · [Client](modules/Simplex/Messaging/Notifications/Client.md) · [Protocol](modules/Simplex/Messaging/Notifications/Protocol.md) + +### Component topology + +![NTF Router — Component Topology](diagrams/ntf-router.svg) + +### Token registration and notification delivery + +```mermaid +sequenceDiagram + participant App + + box NTF Router + participant cl as client thread + participant Store + participant sub as ntfSubscriber + participant push as ntfPush + end + + participant SMP as SMP Router + participant APNS + + App->>cl: TNEW (push token + DH key) + cl->>Store: create token (NTRegistered) + cl->>push: PNVerification (via pushQ) + push->>APNS: verification push + APNS-->>App: verification code (encrypted) + App->>cl: TVFY (code) + cl->>Store: token -> NTActive + + App->>cl: SNEW (subscribe to SMP queue) + cl->>Store: create subscription + cl->>SMP: NKEY (subscribe for notifications) + SMP->>cl: OK (notifier ID) + + Note over SMP: message arrives on queue + SMP->>sub: NMSG (via msgQ) + sub->>Store: update tokenLastNtfs + sub->>push: PNMessage (via pushQ) + push->>APNS: push notification + APNS-->>App: notification (ID only) + App->>SMP: connect and retrieve message +```