diff --git a/README.md b/README.md index b138c8cc9..8d6333f3a 100644 --- a/README.md +++ b/README.md @@ -3,84 +3,78 @@ [![GitHub build](https://github.com/simplex-chat/simplexmq/actions/workflows/build.yml/badge.svg)](https://github.com/simplex-chat/simplexmq/actions/workflows/build.yml) [![GitHub release](https://img.shields.io/github/v/release/simplex-chat/simplexmq)](https://github.com/simplex-chat/simplexmq/releases) -📢 SimpleXMQ v1 is released - with many security, privacy and efficiency improvements, new functionality - see [release notes](https://github.com/simplex-chat/simplexmq/releases/tag/v1.0.0). +## SimpleX Network software -**Please note**: v1 is not backwards compatible, but it has the version negotiation built into all protocol layers for forwards compatibility of this version and backwards compatibility of the future versions, that will be backwards compatible for at least two versions back. +SimpleXMQ provides the software for [SimpleX Network](./protocol/overview-tjr.md) — a general-purpose packet routing network where endpoints exchange data through independently operated routers using resource-based addressing. Unlike IP networks, SimpleX addresses identify resources on routers (queues, data packets), not endpoint devices. Participants do not need globally unique identifiers to communicate. -If you have a server deployed please deploy a new server to a new host and retire the previous version once it is no longer used. +The software is organized in three layers: -## Message broker for unidirectional (simplex) queues - -SimpleXMQ is a message broker for managing message queues and sending messages over public network. It consists of SMP server, SMP client library and SMP agent that implement [SMP protocol](./protocol/simplex-messaging.md) for client-server communication and [SMP agent protocol](./protocol/agent-protocol.md) to manage duplex connections via simplex queues on multiple SMP servers. - -SMP protocol is inspired by [Redis serialization protocol](https://redis.io/topics/protocol), but it is much simpler - it currently has only 10 client commands and 8 server responses. - -SimpleXMQ is implemented in Haskell - it benefits from robust software transactional memory (STM) and concurrency primitives that Haskell provides. - -## SimpleXMQ roadmap - -- SimpleX service protocol and application template - to enable users building services and chat bots that work over SimpleX protocol stack. The first such service will be a notification service for a mobile app. -- SMP queue redundancy and rotation in SMP agent connections. -- SMP agents synchronization to share connections and messages between multiple agents (it would allow using multiple devices for [simplex-chat](https://github.com/simplex-chat/simplex-chat)). - -## Components - -### SMP server - -[SMP server](./apps/smp-server/Main.hs) can be run on any Linux distribution, including low power/low memory devices. OpenSSL library is required for initialization. - -To initialize the server use `smp-server init -n ` (or `smp-server init --ip ` for IP based address) command - it will generate keys and certificates for TLS transport. The fingerprint of offline certificate is used as part of the server address to protect client/server connection against man-in-the-middle attacks: `smp://@[:5223]`. - -SMP server uses in-memory persistence with an optional append-only log of created queues that allows to re-start the server without losing the connections. This log is compacted on every server restart, permanently removing suspended and removed queues. - -To enable store log, initialize server using `smp-server -l` command, or modify `smp-server.ini` created during initialization (uncomment `enable: on` option in the store log section). Use `smp-server --help` for other usage tips. - -Starting from version 2.3.0, when store log is enabled, the server would also enable saving undelivered messages on exit and restoring them on start. This can be disabled via a separate setting `restore_messages` in `smp-server.ini` file. Saving messages would only work if the server is stopped with SIGINT signal (keyboard interrupt), if it is stopped with SIGTERM signal the messages would not be saved. - -> **Please note:** On initialization SMP server creates a chain of two certificates: a self-signed CA certificate ("offline") and a server certificate used for TLS handshake ("online"). **You should store CA certificate private key securely and delete it from the server. If server TLS credential is compromised this key can be used to sign a new one, keeping the same server identity and established connections.** CA private key location by default is `/etc/opt/simplex/ca.key`. - -SMP server implements [SMP protocol](./protocol/simplex-messaging.md). - -#### Running SMP server on MacOS - -SMP server requires OpenSSL library for initialization. On MacOS OpenSSL library may be replaced with LibreSSL, which doesn't support required algorithms. Before initializing SMP server verify you have OpenSSL installed: - -```sh -openssl version +``` + Application (e.g. SimpleX Chat) ++----------------------------------+ +| SimpleX Agent | Layer 3 — bidirectional connections, e2e encryption ++----------------------------------+ +| SimpleX Client Libraries | Layer 2 — protocol clients for SMP, XFTP ++----------------------------------+ +| SimpleX Routers | Layer 1 — network infrastructure (SMP, XFTP, NTF) ++----------------------------------+ ``` -If it says "LibreSSL", please install original OpenSSL: +[SimpleX Chat](https://github.com/simplex-chat/simplex-chat) is one application built on Layer 3. IoT devices, AI services, monitoring systems, and automated services are other applications that can use Layers 2 or 3 directly. -```sh -brew update -brew install openssl -echo 'PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"' >> ~/.zprofile # or follow whatever instructions brew suggests -. ~/.zprofile # or restart your terminal to start a new session -``` +SimpleXMQ is implemented in Haskell, benefiting from robust software transactional memory (STM) and concurrency primitives. -Now `openssl version` should be saying "OpenSSL". You can now run `smp-server init` to initialize your SMP server. +See the [SimpleX Network overview](./protocol/overview-tjr.md) for the full protocol architecture, trust model, and security analysis. -### SMP client library +## Architecture -[SMP client](./src/Simplex/Messaging/Client.hs) is a Haskell library to connect to SMP servers that allows to: +### SimpleX Routers -- execute commands with a functional API. -- receive messages and other notifications via STM queue. -- automatically send keep-alive commands. +Routers are the network infrastructure — they accept, buffer, and deliver packets. Three router types serve different purposes: -### SMP agent +- **SMP routers** provide messaging queues — unidirectional, ordered sequences of fixed-size packets (16,384 bytes). Protocol: [SMP](./protocol/simplex-messaging.md). +- **XFTP routers** provide data packet storage — individually addressed blocks in fixed sizes (64KB–4MB) for larger payloads. Protocol: [XFTP](./protocol/xftp.md). +- **NTF routers** bridge to platform push services (APNS) for mobile notification delivery. Protocol: [Push Notifications](./protocol/push-notifications.md). -[SMP agent library](./src/Simplex/Messaging/Agent.hs) can be used to run SMP agent as part of another application and to communicate with the agent via STM queues, without serializing and parsing commands and responses. +#### Running an SMP router -Haskell type [ACommand](./src/Simplex/Messaging/Agent/Protocol.hs) represents SMP agent protocol to communicate via STM queues. +[SMP server](./apps/smp-server/Main.hs) runs on any Linux distribution. OpenSSL is required for initialization. -See [simplex-chat](https://github.com/simplex-chat/simplex-chat) terminal UI for the example of integrating SMP agent into another application. +Initialize: `smp-server init -n ` (or `--ip `). This generates TLS certificates. The CA certificate fingerprint becomes part of the server address: `smp://@[:5223]`. -[SMP agent executable](./apps/smp-agent/Main.hs) can be used to run a standalone SMP agent process that implements plaintext [SMP agent protocol](./protocol/agent-protocol.md) via TCP port 5224, so it can be used via telnet. It can be deployed in private networks to share access to the connections between multiple applications and services. +The server uses in-memory persistence with an optional append-only store log for queue persistence across restarts. Enable with `smp-server init -l` or in `smp-server.ini`. The log is compacted on every restart. -## Using SMP server and SMP agent +When store log is enabled, undelivered messages are saved on exit (SIGINT only, not SIGTERM) and restored on start. Control this independently with the `restore_messages` setting. -You can either run your own SMP server locally or deploy using [Linode StackScript](https://cloud.linode.com/stackscripts/748014), or try local SMP agent with the deployed servers: +> **Please note:** On initialization, SMP server creates a certificate chain: a self-signed CA certificate ("offline") and a server certificate for TLS ("online"). **Store the CA private key securely and delete it from the server.** If the server TLS credential is compromised, this key can sign a new one while keeping the same server identity. Default location: `/etc/opt/simplex/ca.key`. + +See [docs/ROUTERS.md](./docs/ROUTERS.md) for XFTP/NTF router setup, advanced configuration, MacOS notes, and all deployment options (Docker, installation script, building from source, Linode, DigitalOcean). + +### SimpleX Client Libraries + +[Client libraries](./docs/CLIENT.md) provide low-level protocol access to SimpleX routers. They implement the wire protocols (SMP, XFTP) and handle connection lifecycle, command authentication, and keep-alive. + +The [SMP client](./src/Simplex/Messaging/Client.hs) offers a functional Haskell API with STM queues for asynchronous event delivery. The [XFTP client](./src/Simplex/FileTransfer/Client.hs) handles data packet upload/download with per-download forward secrecy. + +Applications that manage their own encryption and connection logic — IoT devices, sensors, simple data pipelines — can use this layer directly. See [docs/CLIENT.md](./docs/CLIENT.md). + +### SimpleX Agent + +The [Agent](./docs/AGENT.md) builds bidirectional encrypted connections on top of the client libraries. It manages: + +- Duplex connections from unidirectional queue pairs +- End-to-end encryption with double ratchet and post-quantum extensions +- File transfer with chunking, encryption, and multi-router distribution +- Queue rotation for metadata privacy +- Push notification subscriptions + +The [Agent library](./src/Simplex/Messaging/Agent.hs) communicates via STM queues using the [ACommand](./src/Simplex/Messaging/Agent/Protocol.hs) type — no serialization needed. + +See [docs/AGENT.md](./docs/AGENT.md). + +## Quick start + +Public SMP routers for testing: `smp://u2dS9sG8nMNURyZwqASV4yROM28Er0luVTx5X1CsMrU=@smp4.simplex.im` @@ -88,233 +82,16 @@ You can either run your own SMP server locally or deploy using [Linode StackScri `smp://PQUV2eL0t7OStZOoAsPEV2QYWt4-xilbakvGUGOItUo=@smp6.simplex.im` -It's the easiest to try SMP agent via a prototype [simplex-chat](https://github.com/simplex-chat/simplex-chat) terminal UI. +## Deploy routers -## Deploy SMP/XFTP servers on Linux - -You can run your SMP/XFTP server as a Linux process, optionally using a service manager for booting and restarts. - -Notice that `smp-server` and `xftp-server` requires `openssl` as run-time dependency (it is used to generate server certificates during initialization). Install it with your packet manager: +You can run SMP/XFTP routers on any Linux distribution. OpenSSL is required: ```sh -# For Ubuntu +# Ubuntu apt update && apt install openssl ``` -### Install binaries - -#### Using Docker - -On Linux, you can deploy smp and xftp server using Docker. This will download image from [Docker Hub](https://hub.docker.com/r/simplexchat). - -1. Create directories for persistent Docker configuration: - - ```sh - mkdir -p $HOME/simplex/{xftp,smp}/{config,logs} && mkdir -p $HOME/simplex/xftp/files - ``` - -2. Run your Docker container. - - - `smp-server` - - You must change **your_ip_or_domain**. `-e "pass=password"` is optional variable to password-protect your `smp` server: - ```sh - docker run -d \ - -e "ADDR=your_ip_or_domain" \ - -e "PASS=password" \ - -p 5223:5223 \ - -v $HOME/simplex/smp/config:/etc/opt/simplex:z \ - -v $HOME/simplex/smp/logs:/var/opt/simplex:z \ - simplexchat/smp-server:latest - ``` - - - `xftp-server` - - You must change **your_ip_or_domain** and **maximum_storage**. - ```sh - docker run -d \ - -e "ADDR=your_ip_or_domain" \ - -e "QUOTA=maximum_storage" \ - -p 443:443 \ - -v $HOME/simplex/xftp/config:/etc/opt/simplex-xftp:z \ - -v $HOME/simplex/xftp/logs:/var/opt/simplex-xftp:z \ - -v $HOME/simplex/xftp/files:/srv/xftp:z \ - simplexchat/xftp-server:latest - ``` - -#### Using installation script - -**Please note** that currently, only Ubuntu distribution is supported. - -You can install and setup servers automatically using our script: - -```sh -curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/simplex-chat/simplexmq/stable/install.sh -o simplex-server-install.sh &&\ -if echo '53fcdb4ceab324316e2c4cda7e84dbbb344f32550a65975a7895425e5a1be757 simplex-server-install.sh' | sha256sum -c; then - chmod +x ./simplex-server-install.sh - ./simplex-server-install.sh - rm ./simplex-server-install.sh -else - echo "SHA-256 checksum is incorrect!" - rm ./simplex-server-install.sh -fi -``` - -### Build from source - -#### Using Docker - -> **Please note:** to build the app use source code from [stable branch](https://github.com/simplex-chat/simplexmq/tree/stable). - -On Linux, you can build smp server using Docker. - -1. Build your images: - - ```sh - git clone https://github.com/simplex-chat/simplexmq - cd simplexmq - git checkout stable - DOCKER_BUILDKIT=1 docker build -t local/smp-server --build-arg APP="smp-server" --build-arg APP_PORT="5223" . # For xmp-server - DOCKER_BUILDKIT=1 docker build -t local/xftp-server --build-arg APP="xftp-server" --build-arg APP_PORT="443" . # For xftp-server - ``` - -2. Create directories for persistent Docker configuration: - - ```sh - mkdir -p $HOME/simplex/{xftp,smp}/{config,logs} && mkdir -p $HOME/simplex/xftp/files - ``` - -3. Run your Docker container. - - - `smp-server` - - You must change **your_ip_or_domain**. `-e "pass=password"` is optional variable to password-protect your `smp` server: - ```sh - docker run -d \ - -e "ADDR=your_ip_or_domain" \ - -e "PASS=password" \ - -p 5223:5223 \ - -v $HOME/simplex/smp/config:/etc/opt/simplex:z \ - -v $HOME/simplex/smp/logs:/var/opt/simplex:z \ - simplexchat/smp-server:latest - ``` - - - `xftp-server` - - You must change **your_ip_or_domain** and **maximum_storage**. - ```sh - docker run -d \ - -e "ADDR=your_ip_or_domain" \ - -e "QUOTA=maximum_storage" \ - -p 443:443 \ - -v $HOME/simplex/xftp/config:/etc/opt/simplex-xftp:z \ - -v $HOME/simplex/xftp/logs:/var/opt/simplex-xftp:z \ - -v $HOME/simplex/xftp/files:/srv/xftp:z \ - simplexchat/xftp-server:latest - ``` - -#### Using your distribution - -1. Install dependencies and build tools (`GHC`, `cabal` and dev libs): - - ```sh - # On Ubuntu. Depending on your distribution, use your package manager to determine package names. - sudo apt-get update && apt-get install -y build-essential curl libffi-dev libffi7 libgmp3-dev libgmp10 libncurses-dev libncurses5 libtinfo5 pkg-config zlib1g-dev libnuma-dev libssl-dev - export BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3 - export BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.3.0 - curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh - ghcup set ghc "${BOOTSTRAP_HASKELL_GHC_VERSION}" - ghcup set cabal "${BOOTSTRAP_HASKELL_CABAL_VERSION}" - source ~/.ghcup/env - ``` - -2. Build the project: - - ```sh - git clone https://github.com/simplex-chat/simplexmq - cd simplexmq - git checkout stable - cabal update - cabal build exe:smp-server exe:xftp-server - ``` - -3. List compiled binaries: - - `smp-server` - ```sh - cabal list-bin exe:smp-server - ``` - - `xftp-server` - ```sh - cabal list-bin exe:xftp-server - ``` - -- Initialize SMP server with `smp-server init [-l] -n ` or `smp-server init [-l] --ip ` - depending on how you initialize it, either FQDN or IP will be used for server's address. - -- Run `smp-server start` to start SMP server, or you can configure a service manager to run it as a service. - -- Optionally, `smp-server` can be setup for having an onion address in `tor` network. See: [`scripts/tor`](./scripts/tor/). In this case, the server address can have both public and onion hostname pointing to the same server, to allow two people connect when only one of them is using Tor. The server address would be: `smp://@,` - -See [this section](#smp-server) for more information. Run `smp-server -h` and `smp-server init -h` for explanation of commands and options. - -[Linode](https://cloud.linode.com/stackscripts/748014) - -## Deploy SMP server on Linode - -\* You can use free credit Linode offers when [creating a new account](https://www.linode.com/) to deploy an SMP server. - -Deployment on Linode is performed via StackScripts, which serve as recipes for Linode instances, also called Linodes. To deploy SMP server on Linode: - -- Create a Linode account or login with an already existing one. -- Open [SMP server StackScript](https://cloud.linode.com/stackscripts/748014) and click "Deploy New Linode". -- You can optionally configure the following parameters: - - SMP Server store log flag for queue persistence on server restart, recommended. - - [Linode API token](https://www.linode.com/docs/guides/getting-started-with-the-linode-api#get-an-access-token) to attach server address etc. as tags to Linode and to add A record to your 2nd level domain (e.g. `example.com` [domain should be created](https://cloud.linode.com/domains/create) in your account prior to deployment). The API token access scopes: - - read/write for "linodes" - - read/write for "domains" - - Domain name to use instead of Linode IP address, e.g. `smp1.example.com`. -- Choose the region and plan, Shared CPU Nanode with 1Gb is sufficient. -- Provide ssh key to be able to connect to your Linode via ssh. If you haven't provided a Linode API token this step is required to login to your Linode and get the server's fingerprint either from the welcome message or from the file `/etc/opt/simplex/fingerprint` after server starts. See [Linode's guide on ssh](https://www.linode.com/docs/guides/use-public-key-authentication-with-ssh/) . -- Deploy your Linode. After it starts wait for SMP server to start and for tags to appear (if a Linode API token was provided). It may take up to 5 minutes depending on the connection speed on the Linode. Connecting Linode IP address to provided domain name may take some additional time. -- Get `address` and `fingerprint` either from Linode tags (click on a tag and copy it's value from the browser search panel) or via ssh. -- Great, your own SMP server is ready! If you provided FQDN use `smp://@` as SMP server address in the client, otherwise use `smp://@`. - -Please submit an [issue](https://github.com/simplex-chat/simplexmq/issues) if any problems occur. - -[DigitalOcean](https://marketplace.digitalocean.com/apps/simplex-server) - -## Deploy SMP server on DigitalOcean - -> 🚧 DigitalOcean snapshot is currently not up to date, it will soon be updated 🏗️ - -\* When creating a DigitalOcean account you can use [this link](https://try.digitalocean.com/freetrialoffer/) to get free credit. (You would still be required either to provide your credit card details or make a confirmation pre-payment with PayPal) - -To deploy SMP server use [SimpleX Server 1-click app](https://marketplace.digitalocean.com/apps/simplex-server) from DigitalOcean marketplace: - -- Create a DigitalOcean account or login with an already existing one. -- Click 'Create SimpleX server Droplet' button. -- Choose the region and plan according to your requirements (Basic plan should be sufficient). -- Finalize Droplet creation. -- Open "Console" on your Droplet management page to get SMP server fingerprint - either from the welcome message or from `/etc/opt/simplex/fingerprint`. Alternatively you can manually SSH to created Droplet, see [DigitalOcean instruction](https://docs.digitalocean.com/products/droplets/how-to/connect-with-ssh/). -- Great, your own SMP server is ready! Use `smp://@` as SMP server address in the client. - -Please submit an [issue](https://github.com/simplex-chat/simplexmq/issues) if any problems occur. - -> **Please note:** SMP server uses server address as a Common Name for server certificate generated during initialization. If you would like your server address to be FQDN instead of IP address, you can log in to your Droplet and run the commands below to re-initialize the server. Alternatively you can use [Linode StackScript](https://cloud.linode.com/stackscripts/748014) which allows this parameterization. - -```sh -smp-server delete -smp-server init [-l] -n -``` - -## SMP server design - -![SMP server design](./design/server.svg) - -## SMP agent design - -![SMP agent design](./design/agent2.svg) +See [docs/ROUTERS.md](./docs/ROUTERS.md) for Docker, binary installation, building from source, and cloud deployment (Linode, DigitalOcean). ## License diff --git a/docs/AGENT.md b/docs/AGENT.md new file mode 100644 index 000000000..128edc76b --- /dev/null +++ b/docs/AGENT.md @@ -0,0 +1,75 @@ +# SimpleX Agent + +The SimpleX Agent builds bidirectional encrypted connections on top of [SimpleX client libraries](CLIENT.md). It manages the full lifecycle of secure communication: connection establishment, end-to-end encryption, queue rotation, file transfer, and push notifications. + +This is **Layer 3** of the [SimpleX Network architecture](../protocol/overview-tjr.md). Layer 1 is the routers; Layer 2 is the [client libraries](CLIENT.md) that speak the wire protocols. The Agent adds the connection semantics that applications need. + +**Source**: [`Simplex.Messaging.Agent`](../src/Simplex/Messaging/Agent.hs) + +## Connections + +The Agent turns unidirectional SMP queues into bidirectional connections: + +- **Duplex connections**: each connection uses a pair of SMP queues — one for each direction. The queues can be on different routers chosen independently by each party. +- **Connection establishment**: one party creates a connection and generates an invitation (containing router address, queue ID, and public keys). The invitation is passed out-of-band (QR code, link, etc.). The other party joins by creating a reverse queue and completing the handshake. +- **Connection links**: the Agent supports connection links (long and short) for sharing connection invitations via URLs. Short links use a separate SMP queue to store the full invitation, allowing compact QR codes. +- **Queue rotation**: the Agent periodically rotates the underlying SMP queues, limiting the window for metadata correlation. Rotation is transparent to the application — the connection identity is stable while the underlying queues change. +- **Redundant queues**: connections can use multiple queues for reliability. If one router becomes unreachable, messages flow through the remaining queues. + +## Encryption + +The Agent provides end-to-end encryption with forward secrecy and break-in recovery: + +- **Double ratchet**: messages are encrypted using a double ratchet protocol derived from the Signal protocol. Each message uses a unique key; compromising one key does not reveal past or future messages. +- **Post-quantum extensions**: the ratchet supports hybrid key exchange using SNTRUP761 (a lattice-based KEM) combined with X25519 DH. This provides protection against future quantum computers that could break classical DH. +- **Ratchet synchronization**: if the ratchet state becomes desynchronized (e.g., due to message loss or device restore), the Agent detects this and can negotiate resynchronization with the peer. +- **Per-queue encryption**: in addition to end-to-end encryption, each queue has a separate encryption layer between sender and router, preventing traffic correlation even if TLS is compromised. + +## File Transfer + +The Agent handles file transfer over [XFTP](../protocol/xftp.md) routers: + +- **Chunking**: files are split into chunks, each stored as a data packet on an XFTP router. Chunk sizes are fixed powers of 2 (64KB to 4MB), hiding the actual file size. +- **Client-side encryption**: files are encrypted and padded before upload. The recipient decrypts after downloading all chunks. The encryption key and file metadata are sent through the SMP connection, not through XFTP. +- **Multi-router distribution**: chunks can be uploaded to different XFTP routers, and each chunk can have multiple replicas on different routers for redundancy. +- **Redirect chains**: for metadata privacy, file descriptors can be stored as XFTP data packets themselves, creating an indirection layer between the SMP message and the actual file location. + +## Notifications + +The Agent manages push notification subscriptions for mobile devices: + +- **Token registration**: registers device push tokens with NTF (notification) routers, which bridge to platform push services (APNS). +- **Notification subscriptions**: creates NTF subscriptions for SMP queues so that incoming messages trigger push notifications without requiring persistent connections. +- **Privacy preservation**: push notifications contain only a notification ID, not message content. The device wakes, connects to the SMP router, and retrieves the actual message. + +## Integration + +The Agent is designed to be embedded as a Haskell library: + +- **STM queues**: the application communicates with the Agent via STM queues. Commands go in (`ACommand`), events come out (`AEvent`). No serialization or parsing — direct Haskell values. +- **Async operation**: all network operations are asynchronous. The Agent manages internal worker threads for each router connection, message processing, and background tasks (cleanup, statistics, notification supervision). +- **Background mode**: on mobile platforms, the Agent can run in a reduced mode with only the message receiver active, minimizing resource usage when the app is backgrounded. +- **Dual database backends**: the Agent supports both SQLite (for mobile/desktop) and PostgreSQL (for server deployments) as persistence backends, selected at compile time. + +## Use cases + +- **Chat applications**: [SimpleX Chat](https://github.com/simplex-chat/simplex-chat) is the reference application, using the full Agent API for messaging, file sharing, groups, and calls. +- **Bots and automated services**: services that need bidirectional encrypted communication with SimpleX Chat users or other Agent-based applications. +- **Any application needing secure bidirectional communication** over the SimpleX Network without implementing the connection management, encryption, and queue rotation logic directly. + +## What this layer adds over client libraries + +| Capability | Client (Layer 2) | Agent (Layer 3) | +|---|---|---| +| Queue operations | Direct | Managed transparently | +| Connection model | Unidirectional queues | Bidirectional connections | +| Encryption | Application's responsibility | Double ratchet with PQ extensions | +| File transfer | Raw data packet upload/download | Chunking, encryption, reassembly | +| Identity | Per-queue keys | Per-connection, rotatable | +| Notifications | Not available | NTF router integration | + +## Protocol references + +- [Agent Protocol](../protocol/agent-protocol.md) — duplex connection procedure, message format +- [SimpleX Network overview](../protocol/overview-tjr.md) — architecture, trust model +- [PQDR](../protocol/pqdr.md) — post-quantum double ratchet specification diff --git a/docs/CLIENT.md b/docs/CLIENT.md new file mode 100644 index 000000000..6e2ca0c53 --- /dev/null +++ b/docs/CLIENT.md @@ -0,0 +1,75 @@ +# SimpleX Client Libraries + +SimpleX client libraries provide low-level protocol access to SimpleX routers. They implement the wire protocols ([SMP](../protocol/simplex-messaging.md), [XFTP](../protocol/xftp.md)) and handle connection lifecycle, but leave encryption, identity management, and connection orchestration to the application. + +This is **Layer 2** of the [SimpleX Network architecture](../protocol/overview-tjr.md). Layer 1 is the routers themselves; Layer 3 is the [Agent](AGENT.md), which builds bidirectional encrypted connections on top of these libraries. + +## SMP Client + +**Source**: [`Simplex.Messaging.Client`](../src/Simplex/Messaging/Client.hs) + +The SMP client connects to SMP routers and manages messaging queues — the fundamental addressing primitive of the SimpleX Network. Each queue is a unidirectional, ordered sequence of fixed-size packets (16,384 bytes) with separate cryptographic credentials for sending and receiving. + +### Capabilities + +- **Queue management**: create, secure, subscribe to, and delete queues on any SMP router +- **Message sending and receiving**: send messages to a queue's sender address; receive messages from a queue's recipient address +- **Command authentication**: each queue operation is authenticated with per-queue cryptographic keys (Ed25519, Ed448, or X25519) +- **Keep-alive**: automatic ping loop detects and recovers from half-open connections +- **Proxy forwarding**: send messages through a proxy router via 2-hop onion routing (PRXY/PFWD/RFWD commands), protecting the sender's IP address from the destination router +- **Batched commands**: multiple commands can be sent in a single transmission for efficiency + +### API model + +The client uses a functional Haskell API with STM queues for asynchronous event delivery: + +- **Commands** are sent via `sendProtocolCommand` (single) or `sendBatch` (multiple). Each returns a result synchronously or via timeout. +- **Router events** (incoming messages, subscription notifications) arrive on `msgQ`, an STM `TBQueue` that the application reads from its own thread. +- **Connection lifecycle** is managed automatically: the client maintains send, receive, process, and monitor threads internally. When any thread fails, all are torn down and the `disconnected` callback fires. + +### Router identity + +Routers are identified by the SHA-256 hash of their CA certificate fingerprint, not by hostname. The client validates the full X.509 certificate chain on every TLS connection and compares the CA fingerprint against the expected hash from the queue address. This means a DNS or IP-level attacker who cannot produce the correct certificate is detected at connection time. + +## XFTP Client + +**Source**: [`Simplex.FileTransfer.Client`](../src/Simplex/FileTransfer/Client.hs) + +The XFTP client connects to XFTP routers and manages data packets — individually addressed blocks used for larger payload delivery. Data packets come in fixed sizes (64KB, 256KB, 1MB, 4MB), hiding the actual payload size. + +### Capabilities + +- **Data packet creation**: create data packets on routers with sender, recipient, and optional additional recipient credentials +- **Upload**: send encrypted data in a single HTTP/2 streaming request (command + body) +- **Download**: retrieve data packets with per-download ephemeral Diffie-Hellman key exchange, providing forward secrecy — compromising one download key does not reveal other downloads +- **Acknowledgment and deletion**: recipients acknowledge receipt; senders delete data packets after delivery + +### Size selection + +`prepareChunkSizes` selects data packet sizes using a threshold algorithm: if the remaining payload exceeds 75% of the next larger size, it uses the larger size. This balances storage efficiency against the number of round trips. Single-chunk payloads (e.g., redirect descriptors) can use `singleChunkSize` to verify they fit in one data packet. + +## Use cases + +These libraries are appropriate when the application manages its own encryption and connection logic: + +- **IoT sensor data collection**: a sensor creates an SMP queue and sends readings; a collector subscribes and receives them. The queue address (router + queue ID + keys) is provisioned once, out-of-band. +- **Device control**: a controller sends commands to an actuator's queue. Separate queues for commands and telemetry provide unidirectional isolation. +- **Bulk data delivery**: an application encrypts and chunks a file, uploads data packets to XFTP routers, and shares the packet addresses with the recipient out-of-band. +- **Custom protocols**: any application that needs unidirectional, router-mediated packet delivery without the overhead of the Agent's connection management. + +## What this layer does NOT provide + +The following capabilities require the [Agent](AGENT.md) (Layer 3): + +- **Bidirectional connections** — the Agent pairs two unidirectional queues into a duplex connection +- **End-to-end encryption** — the Agent manages double ratchet with post-quantum extensions +- **File transfer** — the Agent handles chunking, encryption, padding, multi-router upload, and reassembly +- **Queue rotation** — the Agent transparently rotates queues to limit metadata correlation +- **Connection discovery** — connection links, short links, and contact addresses are Agent-level abstractions +- **Push notifications** — notification token management and subscription is Agent-level + +## Protocol references + +- [SimpleX Messaging Protocol](../protocol/simplex-messaging.md) — SMP wire format, commands, and security properties +- [XFTP Protocol](../protocol/xftp.md) — XFTP wire format, data packet lifecycle +- [SimpleX Network overview](../protocol/overview-tjr.md) — architecture, trust model, and design rationale diff --git a/docs/ROUTERS.md b/docs/ROUTERS.md new file mode 100644 index 000000000..3169d480e --- /dev/null +++ b/docs/ROUTERS.md @@ -0,0 +1,194 @@ +# SimpleX Routers — Deployment and Configuration + +SimpleX routers are the network infrastructure of the [SimpleX Network](../protocol/overview-tjr.md). They accept, buffer, and deliver data packets between endpoints. Each router operates independently and can be run by any party on standard computing hardware. + +This document covers deployment and advanced configuration. For an overview of the router architecture and trust model, see the [SimpleX Network overview](../protocol/overview-tjr.md). + +## SMP Router + +The SMP router provides messaging queues — unidirectional, ordered sequences of fixed-size packets (16,384 bytes each). It implements the [SimpleX Messaging Protocol](../protocol/simplex-messaging.md). + +### Advanced configuration + +`smp-server.ini` is created during initialization and controls all runtime behavior. + +**Message persistence**: when store log is enabled (`enable: on`), the server saves undelivered messages on exit and restores them on start. This only works with SIGINT (keyboard interrupt); SIGTERM does not trigger message saving. The `restore_messages` setting can be used to override this behavior independently of the store log setting. + +**Tor onion addresses**: the server can have both a public hostname and an onion hostname, allowing two users to connect when only one is using Tor. Configure as: `smp://@,`. See [`scripts/tor/`](../scripts/tor/) for setup instructions. + +### Running on MacOS + +SMP server requires OpenSSL for initialization. MacOS may ship LibreSSL instead, which doesn't support the required algorithms. + +```sh +openssl version +``` + +If it says "LibreSSL", install OpenSSL: + +```sh +brew update +brew install openssl +echo 'PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"' >> ~/.zprofile +. ~/.zprofile +``` + +## XFTP Router + +The XFTP router provides data packet storage — individually addressed blocks in fixed sizes (64KB, 256KB, 1MB, 4MB). It implements the [XFTP protocol](../protocol/xftp.md). Data packets are used for larger payload delivery (files, media) where SMP queue packet sizes would be inefficient. + +Initialize with `xftp-server init` and configure storage quota in `xftp-server.ini`. + +## NTF Router + +The NTF router bridges SimpleX Network to platform push notification services (APNS). It implements the [Push Notifications protocol](../protocol/push-notifications.md). Mobile clients register push tokens with the NTF router, which subscribes to their SMP queues and sends push notifications when messages arrive. The push notification contains only a notification ID, not message content. + +Initialize with `ntf-server init` and configure APNS credentials in `ntf-server.ini`. + +## Deployment methods + +All routers require `openssl` as a runtime dependency for certificate generation during initialization: + +```sh +# Ubuntu +apt update && apt install openssl +``` + +### Docker (prebuilt images) + +Prebuilt images are available from [Docker Hub](https://hub.docker.com/r/simplexchat). + +1. Create directories for persistent configuration: + + ```sh + mkdir -p $HOME/simplex/{xftp,smp}/{config,logs} && mkdir -p $HOME/simplex/xftp/files + ``` + +2. Run: + + **SMP router** — change `your_ip_or_domain`; `-e "PASS=password"` is optional: + ```sh + docker run -d \ + -e "ADDR=your_ip_or_domain" \ + -e "PASS=password" \ + -p 5223:5223 \ + -v $HOME/simplex/smp/config:/etc/opt/simplex:z \ + -v $HOME/simplex/smp/logs:/var/opt/simplex:z \ + simplexchat/smp-server:latest + ``` + + **XFTP router** — change `your_ip_or_domain` and `maximum_storage`: + ```sh + docker run -d \ + -e "ADDR=your_ip_or_domain" \ + -e "QUOTA=maximum_storage" \ + -p 443:443 \ + -v $HOME/simplex/xftp/config:/etc/opt/simplex-xftp:z \ + -v $HOME/simplex/xftp/logs:/var/opt/simplex-xftp:z \ + -v $HOME/simplex/xftp/files:/srv/xftp:z \ + simplexchat/xftp-server:latest + ``` + +### Installation script (Ubuntu) + +```sh +curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/simplex-chat/simplexmq/stable/install.sh -o simplex-server-install.sh &&\ +if echo '53fcdb4ceab324316e2c4cda7e84dbbb344f32550a65975a7895425e5a1be757 simplex-server-install.sh' | sha256sum -c; then + chmod +x ./simplex-server-install.sh + ./simplex-server-install.sh + rm ./simplex-server-install.sh +else + echo "SHA-256 checksum is incorrect!" + rm ./simplex-server-install.sh +fi +``` + +### Build from source + +#### Using Docker + +Build from the [stable branch](https://github.com/simplex-chat/simplexmq/tree/stable): + +```sh +git clone https://github.com/simplex-chat/simplexmq +cd simplexmq +git checkout stable +DOCKER_BUILDKIT=1 docker build -t local/smp-server --build-arg APP="smp-server" --build-arg APP_PORT="5223" . +DOCKER_BUILDKIT=1 docker build -t local/xftp-server --build-arg APP="xftp-server" --build-arg APP_PORT="443" . +``` + +Then run with the same Docker commands as above, replacing `simplexchat/smp-server:latest` with `local/smp-server` (and similarly for XFTP). + +#### Native build + +1. Install dependencies: + + ```sh + # Ubuntu + sudo apt-get update && apt-get install -y build-essential curl libffi-dev libffi7 libgmp3-dev libgmp10 libncurses-dev libncurses5 libtinfo5 pkg-config zlib1g-dev libnuma-dev libssl-dev + export BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3 + export BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.3.0 + curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh + ghcup set ghc "${BOOTSTRAP_HASKELL_GHC_VERSION}" + ghcup set cabal "${BOOTSTRAP_HASKELL_CABAL_VERSION}" + source ~/.ghcup/env + ``` + +2. Build: + + ```sh + git clone https://github.com/simplex-chat/simplexmq + cd simplexmq + git checkout stable + cabal update + cabal build exe:smp-server exe:xftp-server + ``` + +3. Find binaries: + + ```sh + cabal list-bin exe:smp-server + cabal list-bin exe:xftp-server + ``` + +4. Initialize and run: + + ```sh + smp-server init [-l] -n # or --ip + smp-server start + ``` + +### Linode StackScript + +[Deploy via Linode StackScript](https://cloud.linode.com/stackscripts/748014) — Shared CPU Nanode with 1GB is sufficient. + +Configuration options: +- SMP Server store log flag for queue persistence (recommended) +- [Linode API token](https://www.linode.com/docs/guides/getting-started-with-the-linode-api#get-an-access-token) for automatic DNS and tagging (scopes: read/write for "linodes" and "domains") +- Domain name (e.g., `smp1.example.com`) — the [domain must exist](https://cloud.linode.com/domains/create) in your Linode account + +After deployment (up to 5 minutes), get the server address from Linode tags or SSH: `smp://@`. + +### DigitalOcean 1-click + +[SimpleX Server 1-click app](https://marketplace.digitalocean.com/apps/simplex-server) from DigitalOcean marketplace. + +After deployment, get the fingerprint from the Droplet console (`/etc/opt/simplex/fingerprint`). Server address: `smp://@`. + +To use FQDN instead of IP: + +```sh +smp-server delete +smp-server init [-l] -n +``` + +## Monitoring + +SMP and XFTP routers expose Prometheus metrics via a control port. The control port also supports commands for runtime inspection (queue counts, client counts, statistics). See module specs for details on available metrics and control commands. + +## Protocol references + +- [SimpleX Messaging Protocol](../protocol/simplex-messaging.md) — SMP wire format and security properties +- [XFTP Protocol](../protocol/xftp.md) — data packet protocol +- [Push Notifications Protocol](../protocol/push-notifications.md) — NTF protocol +- [SimpleX Network overview](../protocol/overview-tjr.md) — architecture and trust model