product and engineering specifications for simplexmq

CODE.md

@@ -0,0 +1,210 @@
+# simplexmq — LLM Navigation Guide
+
+This file is the entry point for LLMs working on simplexmq. Read it before making any code changes.
+
+## Three-Layer Architecture
+
+simplexmq maintains three documentation layers alongside source code:
+
+| Layer | Directory | Answers | Audience |
+|-------|-----------|---------|----------|
+| **Product** | `product/` | What does this do? Who uses it? What must never break? | Anyone reasoning about behavior, privacy, security |
+| **Spec** | `spec/` | How does the code work? What does each function do? What are the security invariants? | LLMs and developers modifying code |
+| **Protocol** | `protocol/` | What is the wire protocol? What are the message formats and state machines? | Protocol implementors, formal verification |
+
+Additionally:
+- `rfcs/` — Protocol evolution: each RFC describes a delta to a protocol spec
+- `product/threat-model.md` — Comprehensive threat model across all protocols
+- `spec/security-invariants.md` — Every security invariant with enforcement and test coverage
+
+## Navigation Workflow
+
+When modifying code, follow this sequence:
+
+1. **Identify scope** — Find the relevant component in `product/concepts.md`
+2. **Load product context** — Read the component file in `product/components/` to understand what users depend on
+3. **Load spec context** — Read the relevant `spec/` file(s) for implementation details and call graphs
+4. **Check security** — Read `spec/security-invariants.md` for any invariants enforced by the code you're changing
+5. **Load source** — Read the actual source files referenced in spec/
+6. **Identify impact** — Trace the call graph to understand what your change affects
+7. **Implement** — Make the change
+8. **Update all layers** — Update spec/, product/, and protocol/ (if wire protocol changed) to stay coherent
+
+## Protocol Specifications
+
+Consolidated protocol specs live in `protocol/`. These describe the wire protocols as originally specified. Code has advanced beyond these versions — Phase 2 of this project will synchronize them.
+
+| File | Protocol | Spec version | Code version |
+|------|----------|-------------|--------------|
+| `simplex-messaging.md` | SMP (simplex messaging) | v9 | SMP relay v18, SMP client v4 |
+| `agent-protocol.md` | Agent (duplex connections) | v5 | Agent v7 |
+| `xftp.md` | XFTP (file transfer) | v2 | XFTP v3 |
+| `xrcp.md` | XRCP (remote control) | v1 | RCP v1 |
+| `push-notifications.md` | Push notifications | v2 | NTF v3 |
+| `pqdr.md` | PQDR (post-quantum double ratchet) | v1 | E2E v3 |
+| `overview-tjr.md` | Cross-protocol overview | — | — |
+
+Note: SMP has multiple version axes — `VersionSMP` (relay/transport, currently 18), `VersionSMPC` (client protocol, currently 4), and `VersionSMPA` (agent, currently 7). These are negotiated independently.
+
+Protocol specs are amended in place when implementation changes. RFCs in `rfcs/` track the evolution history.
+
+## Source Structure
+
+```
+src/Simplex/
+  Messaging/
+    Protocol.hs, Protocol/Types.hs      — SMP wire protocol types + encoding
+    Client.hs                            — SMP client (protocol operations, proxy relay)
+    Client/Agent.hs                      — Low-level async SMP agent
+    Server.hs                            — SMP server request handling
+    Server/Env/STM.hs                    — Server environment + STM state
+    Server/Main.hs, Server/Main/Init.hs  — Server CLI + initialization
+    Server/QueueStore/                   — Queue storage (STM, Postgres)
+    Server/MsgStore/                     — Message storage (STM, Journal, Postgres)
+    Server/MsgStore/Journal.hs           — Journal message store (1000 lines)
+    Server/StoreLog/                     — Store log (append-only write, read-compact-rewrite restore)
+    Server/NtfStore.hs                   — Message notification store
+    Server/Control.hs, Server/CLI.hs     — Control protocol + CLI utilities
+    Server/Stats.hs, Server/Prometheus.hs — Metrics
+    Server/Information.hs                — Server public information / metadata
+    Agent.hs                             — SMP agent: duplex connections, queue rotation
+    Agent/Client.hs                      — Agent's SMP/XFTP/NTF client management
+    Agent/Protocol.hs                    — Agent wire protocol types + encoding (2200 lines)
+    Agent/Store.hs                       — Agent storage types (queues, connections, messages)
+    Agent/Store/AgentStore.hs              — Agent storage implementation (3500 lines)
+    Agent/Store/                          — Agent storage backends (SQLite, Postgres)
+    Agent/Env/SQLite.hs                  — Agent environment + configuration
+    Agent/NtfSubSupervisor.hs            — Notification subscription management
+    Agent/TSessionSubs.hs                — Transport session subscriptions
+    Agent/Stats.hs                       — Agent statistics
+    Agent/RetryInterval.hs               — Retry interval logic
+    Agent/Lock.hs                        — Named locks
+    Agent/QueryString.hs                 — Query string parsing
+    Transport.hs                         — TLS transport abstraction + handshake
+    Transport/Client.hs, Transport/Server.hs — TLS client + server
+    Transport/HTTP2.hs                   — HTTP/2 transport setup
+    Transport/HTTP2/Client.hs            — HTTP/2 client
+    Transport/HTTP2/Server.hs            — HTTP/2 server
+    Transport/HTTP2/File.hs              — HTTP/2 file streaming
+    Transport/WebSockets.hs              — WebSocket adapter
+    Transport/Buffer.hs                  — Transport buffering
+    Transport/KeepAlive.hs               — TCP keepalive
+    Transport/Shared.hs                  — Certificate chain validation
+    Transport/Credentials.hs             — TLS credential generation
+    Crypto.hs                            — All cryptographic primitives
+    Crypto/File.hs                       — File encryption (NaCl secret box + lazy)
+    Crypto/Lazy.hs                       — Lazy hashing + encryption
+    Crypto/Ratchet.hs                    — Double ratchet + PQDR
+    Crypto/ShortLink.hs                  — Short link key derivation
+    Crypto/SNTRUP761.hs                  — Post-quantum KEM hybrid secret
+    Crypto/SNTRUP761/Bindings.hs         — sntrup761 C FFI bindings
+    Notifications/Protocol.hs            — NTF wire protocol types + encoding
+    Notifications/Types.hs               — NTF agent types (tokens, subscriptions)
+    Notifications/Transport.hs           — NTF transport handshake
+    Notifications/Client.hs              — NTF client operations
+    Notifications/Server.hs              — NTF server
+    Notifications/Server/Env.hs          — NTF server environment + config
+    Notifications/Server/Store.hs        — NTF server storage (STM)
+    Notifications/Server/Store/Postgres.hs — NTF server storage (Postgres)
+    Notifications/Server/Push/APNS.hs    — Apple push notification integration
+    Notifications/Server/Push/APNS/Internal.hs — APNS HTTP/2 client
+    Notifications/Server/Main.hs         — NTF server CLI
+    Notifications/Server/Stats.hs        — NTF server metrics
+    Notifications/Server/Prometheus.hs   — NTF Prometheus metrics
+    Notifications/Server/Control.hs      — NTF server control
+    Encoding.hs, Encoding/String.hs      — Binary + string encoding
+    Version.hs, Version/Internal.hs      — Version ranges + negotiation
+    Util.hs                              — Utilities (error handling, STM, grouping)
+    Parsers.hs                           — Attoparsec parser combinators
+    TMap.hs                              — Transactional map (STM)
+    Compression.hs                       — Zstd compression
+    ServiceScheme.hs                     — Service scheme + server location types
+    Session.hs                           — Session variables (TVar-based)
+    SystemTime.hs                        — Rounded system time types
+  FileTransfer/
+    Protocol.hs                          — XFTP wire protocol types + encoding
+    Client.hs                            — XFTP client operations
+    Client/Agent.hs                      — XFTP client agent (connection pooling)
+    Client/Main.hs                       — XFTP CLI client implementation
+    Client/Presets.hs                    — Default XFTP servers
+    Server.hs                            — XFTP server request handling
+    Server/Env.hs                        — XFTP server environment + config
+    Server/Store.hs                      — XFTP server storage
+    Server/StoreLog.hs                   — XFTP server store log
+    Server/Main.hs                       — XFTP server CLI
+    Server/Stats.hs                      — XFTP server metrics
+    Server/Prometheus.hs                 — XFTP Prometheus metrics
+    Server/Control.hs                    — XFTP server control
+    Agent.hs                             — XFTP agent operations
+    Description.hs                       — File description format
+    Transport.hs                         — XFTP transport
+    Crypto.hs                            — File encryption for transfer
+    Types.hs                             — File transfer types
+    Chunks.hs                            — Chunk sizing
+  RemoteControl/
+    Client.hs                            — XRCP client (ctrl device)
+    Invitation.hs                        — XRCP invitation handling
+    Discovery.hs                         — Local network discovery
+    Discovery/Multicast.hsc              — Multicast discovery (C FFI)
+    Types.hs                             — XRCP types + version
+
+apps/
+  smp-server/Main.hs                    — SMP server executable
+  smp-server/web/Static.hs              — SMP server web static files
+  xftp-server/Main.hs                   — XFTP server executable
+  xftp/Main.hs                          — XFTP CLI executable
+  ntf-server/Main.hs                    — Notification server executable
+  smp-agent/Main.hs                     — SMP agent (experimental, not in cabal)
+```
+
+## Linking Conventions
+
+### spec → src
+Fully qualified exported function names inline in prose: `Simplex.Messaging.Client.connectSMPProxiedRelay`. Use Grep/Glob to locate in source. For app targets: `xftp/Main.main`.
+
+### src → spec
+Comment above function:
+```haskell
+-- spec/crypto-tls.md#certificate-chain-validation
+-- Validates relay certificate chain to prevent proxy MITM (SI-XX)
+connectSMPProxiedRelay :: ...
+```
+
+### spec ↔ spec
+Named markdown heading anchors: `spec/crypto.md#ed25519-signing`
+
+### spec ↔ product
+Cross-references: `product/rules.md#pr-05`, `spec/security-invariants.md#si-01`
+
+### protocol/ references
+`protocol/simplex-messaging.md` with section name
+
+## Build Flags
+
+simplexmq builds with several flag combinations:
+
+| Flag | Effect |
+|------|--------|
+| (none) | Default: SQLite storage, all executables |
+| `-fserver_postgres` | Postgres backend for SMP server |
+| `-fclient_postgres` | Postgres backend for agent storage |
+| `-fclient_library` | Library-only build (no server executables) |
+| `-fswift` | Swift JSON format for mobile bindings |
+| `-fuse_crypton` | Use crypton in cryptostore |
+
+All flag combinations must compile with `--enable-tests`. Verify with:
+```
+cabal build all --ghc-options="-O0" [-flags] [--enable-tests]
+```
+
+## Change Protocol
+
+Every code change must maintain coherence across all three layers:
+
+1. **Code change** — Implement in src/
+2. **Spec update** — Update the relevant spec/ file(s): types, call graphs, security notes
+3. **Product update** — If user-visible behavior changed, update product/ files
+4. **Protocol update** — If wire protocol changed, amend protocol/ spec (requires user approval)
+5. **Security check** — If the change touches a trust boundary, update spec/security-invariants.md
+
+Protocol spec amendments require explicit user approval before committing.

product/README.md

@@ -0,0 +1,22 @@
+# SimpleX Network — Product Layer
+
+> What does this do? Who uses it? What must never break?
+
+## Vision
+
+<!-- Extracted from protocol specs and philosophy -->
+
+## Components
+
+| Component | Description | Spec | Protocol |
+|-----------|-------------|------|----------|
+| SMP | Simplex messaging queues | spec/smp-protocol.md | protocol/simplex-messaging.md |
+| Agent | Duplex connections over simplex queues | spec/agent.md | protocol/agent-protocol.md |
+| XFTP | File transfer via encrypted chunks | spec/xftp-protocol.md | protocol/xftp.md |
+| XRCP | Remote control of mobile clients | spec/remote-control.md | protocol/xrcp.md |
+| NTF | Push notifications with privacy | spec/ntf-protocol.md | protocol/push-notifications.md |
+| Servers | SMP, XFTP, NTF server operation | spec/smp-server.md | — |
+
+## Capability Map
+
+<!-- Per-component: what users can do, what the system guarantees -->

product/components/agent.md

@@ -0,0 +1,9 @@
+# Agent — Duplex Connections
+
+> Bidirectional connections built over pairs of simplex queues.
+
+## Users
+
+## Connection Lifecycle
+
+## Guarantees

product/components/notifications.md

@@ -0,0 +1,7 @@
+# Push Notifications
+
+> Push notifications with metadata privacy.
+
+## Users
+
+## Privacy Trade-offs

product/components/servers.md

@@ -0,0 +1,9 @@
+# Server Operation
+
+> SMP, XFTP, and NTF server deployment and operation.
+
+## Deployment
+
+## Configuration
+
+## Monitoring

product/components/smp.md

@@ -0,0 +1,9 @@
+# SMP — Simplex Messaging Protocol
+
+> Unidirectional messaging queues with sender/receiver separation.
+
+## Users
+
+## Guarantees
+
+## Privacy Properties

product/components/xftp.md

@@ -0,0 +1,9 @@
+# XFTP — File Transfer
+
+> Encrypted file transfer via content-addressed chunks.
+
+## Users
+
+## Guarantees
+
+## Privacy Properties

product/components/xrcp.md

@@ -0,0 +1,7 @@
+# XRCP — Remote Control
+
+> Remote control of mobile clients from desktop.
+
+## Users
+
+## Trust Model

product/concepts.md

@@ -0,0 +1,5 @@
+# Concepts & Entity Index
+
+> Domain concepts with cross-references to spec/ and src/.
+
+<!-- Each concept: name, definition, where it appears in spec/ and src/ -->

product/glossary.md

@@ -0,0 +1,5 @@
+# Glossary
+
+> Domain terminology used across simplexmq.
+
+<!-- Each term: name, definition, where used -->

product/goals.md

@@ -0,0 +1,5 @@
+# Design Goals
+
+> Verified against protocol specs and code.
+
+<!-- Each goal: statement, source (which protocol spec), verification status -->

product/rules.md

@@ -0,0 +1,5 @@
+# Invariant Rules
+
+> Invariants users depend on: privacy, delivery, ordering, security.
+
+<!-- Each rule: ID (PR-XX), statement, rationale, enforcement (qualified function name), test coverage -->

product/threat-model.md

@@ -0,0 +1,22 @@
+# Threat Model
+
+> Comprehensive threat model across all protocols.
+
+Consistent with threat models in:
+- `protocol/overview-tjr.md` (cross-protocol)
+- `protocol/simplex-messaging.md` (SMP)
+- `protocol/xftp.md` (XFTP)
+- `protocol/xrcp.md` (XRCP)
+- `protocol/push-notifications.md` (notifications)
+
+## Actors
+
+<!-- Threat actors and their capabilities -->
+
+## Trust Boundaries
+
+<!-- Per-protocol trust boundaries -->
+
+## Security Properties
+
+<!-- Properties the system must maintain -->

spec/README.md

@@ -0,0 +1,64 @@
+# Spec Layer
+
+> How does the code work? What does each function do? What are the security invariants?
+
+## Conventions
+
+Each spec file documents:
+1. **Purpose** — What this component does
+2. **Protocol reference** — Link to `protocol/` file (where applicable)
+3. **Types** — Key data types with field descriptions
+4. **Functions** — Every exported function with call graph
+5. **Security notes** — Trust assumptions, validation requirements
+
+Function documentation format:
+```
+### Module.functionName
+**Purpose**: ...
+**Calls**: Module.a, Module.b
+**Called by**: Module.c
+**Invariant**: SI-XX
+**Security**: ...
+```
+
+## Index
+
+### Protocol Implementation
+- [smp-protocol.md](smp-protocol.md) — SMP commands, types, encoding
+- [xftp-protocol.md](xftp-protocol.md) — XFTP commands, chunk operations
+- [ntf-protocol.md](ntf-protocol.md) — NTF commands, token/subscription lifecycle
+- [xrcp-protocol.md](xrcp-protocol.md) — XRCP session handshake, commands
+- [agent-protocol.md](agent-protocol.md) — Agent connection procedures, queue rotation
+
+### Cryptography
+- [crypto.md](crypto.md) — All primitives: Ed25519, X25519, NaCl, AES-GCM, SHA, HKDF
+- [crypto-ratchet.md](crypto-ratchet.md) — Double ratchet + PQDR
+- [crypto-tls.md](crypto-tls.md) — TLS setup, certificate chains, validation
+
+### Transport
+- [transport.md](transport.md) — Transport abstraction, handshake, block padding
+- [transport-http2.md](transport-http2.md) — HTTP/2 framing, file streaming
+- [transport-websocket.md](transport-websocket.md) — WebSocket adapter
+
+### Server Implementations
+- [smp-server.md](smp-server.md) — SMP server
+- [xftp-server.md](xftp-server.md) — XFTP server
+- [ntf-server.md](ntf-server.md) — Notification server
+
+### Client Implementations
+- [smp-client.md](smp-client.md) — SMP client, proxy relay
+- [xftp-client.md](xftp-client.md) — XFTP client
+- [agent.md](agent.md) — SMP agent, duplex connections
+
+### Storage
+- [storage-server.md](storage-server.md) — Server storage backends
+- [storage-agent.md](storage-agent.md) — Agent storage backends
+
+### Auxiliary
+- [encoding.md](encoding.md) — Binary and string encoding
+- [version.md](version.md) — Version ranges and negotiation
+- [remote-control.md](remote-control.md) — XRCP implementation
+- [compression.md](compression.md) — Zstd compression
+
+### Security
+- [security-invariants.md](security-invariants.md) — All security invariants

spec/agent-protocol.md

@@ -0,0 +1,13 @@
+# Agent Protocol Implementation
+
+> Implements agent connection procedures, queue rotation, and duplex messaging.
+
+**Protocol reference**: [`protocol/agent-protocol.md`](../protocol/agent-protocol.md)
+
+## Types
+
+## Connection Procedures
+
+## Queue Rotation
+
+## Functions

spec/agent.md

@@ -0,0 +1,13 @@
+# SMP Agent
+
+> SMP agent implementation: duplex connections, queue rotation, ratchet sync, and notification subscriptions.
+
+## Duplex Connections
+
+## Queue Rotation
+
+## Ratchet Sync
+
+## Notification Subscriptions
+
+## Functions

spec/compression.md

@@ -0,0 +1,7 @@
+# Compression
+
+> Compression support for SimpleX protocols.
+
+## Zstd
+
+## Functions

spec/crypto-ratchet.md

@@ -0,0 +1,13 @@
+# Double Ratchet & PQDR
+
+> Implements the double ratchet algorithm with post-quantum extensions (PQDR).
+
+**Protocol reference**: [`protocol/pqdr.md`](../protocol/pqdr.md)
+
+## State
+
+## Transitions
+
+## Key Derivation
+
+## Functions

spec/crypto-tls.md

@@ -0,0 +1,11 @@
+# TLS & Certificate Chains
+
+> TLS session setup, certificate chain construction, and server identity validation.
+
+## TLS Setup
+
+## Certificate Validation
+
+## Trust Anchoring
+
+## Functions

spec/crypto.md

@@ -0,0 +1,19 @@
+# Cryptographic Primitives
+
+> All cryptographic primitives used across SimpleX protocols.
+
+## Ed25519
+
+## X25519
+
+## NaCl
+
+## AES-GCM
+
+## SHA
+
+## HKDF
+
+## Key Generation
+
+## Functions

spec/encoding.md

@@ -0,0 +1,11 @@
+# Encoding
+
+> Binary and string encoding used across all SimpleX protocols.
+
+## Binary Encoding
+
+## String Encoding
+
+## Parsers
+
+## Functions

spec/ntf-protocol.md

@@ -0,0 +1,15 @@
+# NTF Protocol Implementation
+
+> Implements NTF commands, token registration, and subscription lifecycle for push notifications.
+
+**Protocol reference**: [`protocol/push-notifications.md`](../protocol/push-notifications.md)
+
+## Types
+
+## Commands
+
+## Token Lifecycle
+
+## Subscription Lifecycle
+
+## Functions

spec/ntf-server.md

@@ -0,0 +1,11 @@
+# Notification Server
+
+> Notification server implementation: token management, subscriptions, and APNS integration.
+
+## Token Management
+
+## Subscription Management
+
+## APNS Integration
+
+## Functions

spec/remote-control.md

@@ -0,0 +1,11 @@
+# Remote Control (XRCP)
+
+> XRCP implementation: discovery, invitation, and session management.
+
+## Discovery
+
+## Invitation
+
+## Session Management
+
+## Functions

spec/security-invariants.md

@@ -0,0 +1,19 @@
+# Security Invariants
+
+> Every security invariant with enforcement and test coverage.
+
+## Format
+
+```
+### SI-XX: [Name]
+**Statement**: [Precise invariant]
+**Threat**: [What attack this prevents]
+**Actors**: [Which threat model actors are relevant]
+**Enforced by**: [Qualified function names] — [how]
+**Tested by**: [test module.function] or [MISSING TEST]
+**Product rule**: PR-XX
+```
+
+## Invariants
+
+<!-- Populated in Phase 3.8 -->

spec/smp-client.md

@@ -0,0 +1,11 @@
+# SMP Client
+
+> SMP client implementation: protocol operations, proxy relay, and reconnection logic.
+
+## Protocol Operations
+
+## Proxy Relay
+
+## Reconnection
+
+## Functions

spec/smp-protocol.md

@@ -0,0 +1,13 @@
+# SMP Protocol Implementation
+
+> Implements SMP commands, types, and binary encoding for the SimpleX Messaging Protocol.
+
+**Protocol reference**: [`protocol/simplex-messaging.md`](../protocol/simplex-messaging.md)
+
+## Types
+
+## Commands
+
+## Encoding
+
+## Functions

spec/smp-server.md

@@ -0,0 +1,13 @@
+# SMP Server
+
+> SMP server implementation: connection handling, queue operations, proxying, and control port.
+
+## Connection Handling
+
+## Queue Operations
+
+## Proxying
+
+## Control
+
+## Functions

spec/storage-agent.md

@@ -0,0 +1,11 @@
+# Agent Storage
+
+> Agent storage backends: SQLite, Postgres, and migration framework.
+
+## SQLite Backend
+
+## Postgres Backend
+
+## Migration Framework
+
+## Functions

spec/storage-server.md

@@ -0,0 +1,9 @@
+# Server Storage
+
+> Server storage backends: STM queues and message stores (STM, Journal, Postgres).
+
+## STM Queues
+
+## Message Stores (STM, Journal, Postgres)
+
+## Functions

spec/transport-http2.md

@@ -0,0 +1,13 @@
+# HTTP/2 Transport
+
+> HTTP/2 framing, client and server sessions, and file streaming for XFTP.
+
+## Framing
+
+## Client Sessions
+
+## Server Sessions
+
+## File Streaming
+
+## Functions

spec/transport-websocket.md

@@ -0,0 +1,7 @@
+# WebSocket Transport
+
+> WebSocket adapter for browser-based SimpleX clients.
+
+## Adapter
+
+## Functions

spec/transport.md

@@ -0,0 +1,11 @@
+# Transport Layer
+
+> Transport abstraction, handshake protocol, and block padding for metadata privacy.
+
+## Abstraction
+
+## Handshake Protocol
+
+## Block Padding
+
+## Functions

spec/version.md

@@ -0,0 +1,9 @@
+# Version Negotiation
+
+> Version ranges and compatibility checking for protocol evolution.
+
+## Version Ranges
+
+## Compatibility
+
+## Functions

spec/xftp-client.md

@@ -0,0 +1,11 @@
+# XFTP Client
+
+> XFTP client implementation: file operations, CLI interface, and agent integration.
+
+## File Operations
+
+## CLI
+
+## Agent
+
+## Functions

spec/xftp-protocol.md

@@ -0,0 +1,13 @@
+# XFTP Protocol Implementation
+
+> Implements XFTP commands, types, and chunk operations for the SimpleX File Transfer Protocol.
+
+**Protocol reference**: [`protocol/xftp.md`](../protocol/xftp.md)
+
+## Types
+
+## Commands
+
+## Chunk Operations
+
+## Functions

spec/xftp-server.md

@@ -0,0 +1,11 @@
+# XFTP Server
+
+> XFTP server implementation: chunk storage, recipient management, and control port.
+
+## Chunk Storage
+
+## Recipient Management
+
+## Control
+
+## Functions

spec/xrcp-protocol.md

@@ -0,0 +1,13 @@
+# XRCP Protocol Implementation
+
+> Implements XRCP session handshake and commands for remote control of SimpleX clients.
+
+**Protocol reference**: [`protocol/xrcp.md`](../protocol/xrcp.md)
+
+## Types
+
+## Session Handshake
+
+## Commands
+
+## Functions