mirror of
https://github.com/simplex-chat/simplexmq.git
synced 2026-05-25 09:54:29 +00:00
terms 2
This commit is contained in:
+4
-4
@@ -44,15 +44,15 @@
|
||||
|
||||
- **NTF startup resubscription**: `resubscribe` runs as detached `forkIO` (not in `raceAny_` group), uses `mapConcurrently` across SMP routers, each with `subscribeLoop` using 100x database batch multiplier and cursor-based pagination. `ExitCode` exceptions from `exitFailure` on DB error propagate to main thread despite `forkIO`. `getServerNtfSubscriptions` claims subscriptions by batch-updating to `NSPending`. Spans [Server.hs](modules/Simplex/Messaging/Notifications/Server.md), [Store/Postgres.hs](modules/Simplex/Messaging/Notifications/Server/Store/Postgres.md).
|
||||
|
||||
- **XFTP file upload pipeline**: Agent-side encryption (streaming 64KB blocks, fixed-size padding) → chunk size selection (75% threshold algorithm) → per-router chunk creation with ID collision retry (3 attempts) → recipient registration (recursive batching up to `maxRecipients` per FADD) → per-router upload (command + file body in single HTTP/2 streaming request) → file description generation (cross-product: M chunks × R replicas × N recipients → N descriptions). Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md) (worker orchestration, description generation), [Client.hs](modules/Simplex/FileTransfer/Client.md) (upload protocol), [Server.hs](modules/Simplex/FileTransfer/Server.md) (quota reservation with rollback, skipCommitted idempotency), [Crypto.hs](modules/Simplex/FileTransfer/Crypto.md) (streaming encryption with embedded header), [Description.hs](modules/Simplex/FileTransfer/Description.md) (validation, first-replica-only digest optimization).
|
||||
- **XFTP file upload pipeline**: Agent-side encryption (streaming 64KB blocks, fixed-size padding) → chunk size selection (75% threshold algorithm) → per-router data packet creation with ID collision retry (3 attempts) → recipient registration (recursive batching up to `maxRecipients` per FADD) → per-router data packet upload (command + data in single HTTP/2 streaming request) → file description generation (cross-product: M chunks × R replicas × N recipients → N descriptions). Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md) (worker orchestration, description generation), [Client.hs](modules/Simplex/FileTransfer/Client.md) (upload protocol), [Server.hs](modules/Simplex/FileTransfer/Server.md) (quota reservation with rollback, skipCommitted idempotency), [Crypto.hs](modules/Simplex/FileTransfer/Crypto.md) (streaming encryption with embedded header), [Description.hs](modules/Simplex/FileTransfer/Description.md) (validation, first-replica-only digest optimization).
|
||||
|
||||
- **XFTP file download pipeline**: Description parsing (ValidFileDescription validation, YAML or web URI) → per-router chunk download with ephemeral DH key pair per download (forward secrecy) → size and digest verification before decryption → streaming decryption with auth tag verification (output deleted on failure) → redirect resolution (depth-1 chain: decrypt redirect YAML, validate size/digest, download actual file). Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md) (worker orchestration, redirect handling), [Client.hs](modules/Simplex/FileTransfer/Client.md) (ephemeral DH, chunk-proportional timeout), [Client/Main.hs](modules/Simplex/FileTransfer/Client/Main.md) (web URI decoding, parallel download with router grouping), [Crypto.hs](modules/Simplex/FileTransfer/Crypto.md) (dual decrypt paths, auth tag deletion), [Description.hs](modules/Simplex/FileTransfer/Description.md) (redirect file descriptions).
|
||||
- **XFTP file download pipeline**: Description parsing (ValidFileDescription validation, YAML or web URI) → per-router data packet download with ephemeral DH key pair per download (forward secrecy) → size and digest verification before decryption → streaming decryption with auth tag verification (output deleted on failure) → redirect resolution (depth-1 chain: decrypt redirect YAML, validate size/digest, download actual file). Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md) (worker orchestration, redirect handling), [Client.hs](modules/Simplex/FileTransfer/Client.md) (ephemeral DH, size-proportional timeout), [Client/Main.hs](modules/Simplex/FileTransfer/Client/Main.md) (web URI decoding, parallel download with router grouping), [Crypto.hs](modules/Simplex/FileTransfer/Crypto.md) (dual decrypt paths, auth tag deletion), [Description.hs](modules/Simplex/FileTransfer/Description.md) (redirect file descriptions).
|
||||
|
||||
- **XFTP handshake state machine**: Three-state session-cached handshake (`No entry` → `HandshakeSent` → `HandshakeAccepted`) per HTTP/2 session. Web clients use `xftp-web-hello` header and challenge-response identity proof; native clients use standard ALPN. SNI presence gates CORS headers, web serving, and SESSION error for unrecognized connections. Key reuse on re-hello preserves existing DH keys. Spans [Server.hs](modules/Simplex/FileTransfer/Server.md) (handshake logic, CORS, web serving), [Client.hs](modules/Simplex/FileTransfer/Client.md) (ALPN selection, cert chain validation), [Transport.hs](modules/Simplex/FileTransfer/Transport.md) (block size, version).
|
||||
|
||||
- **XFTP storage lifecycle**: Quota reservation via atomic `stateTVar` before upload → rollback on failure (subtract + delete partial file) → physical file deleted before store cleanup (crash risk: store references missing file) → `RoundedSystemTime 3600` for privacy-preserving expiration timestamps → expiration with configurable throttling (100ms between files) → startup storage reconciliation (override stats from live store). Spans [Server.hs](modules/Simplex/FileTransfer/Server.md), [Server/Store.hs](modules/Simplex/FileTransfer/Server/Store.md), [Server/Env.hs](modules/Simplex/FileTransfer/Server/Env.md), [Server/StoreLog.hs](modules/Simplex/FileTransfer/Server/StoreLog.md) (error-resilient replay, compaction).
|
||||
- **XFTP storage lifecycle**: Quota reservation via atomic `stateTVar` before upload → rollback on failure (subtract + delete partial data packet) → stored data packet deleted before store cleanup (crash risk: store references missing data packet) → `RoundedSystemTime 3600` for privacy-preserving expiration timestamps → expiration with configurable throttling (100ms between data packets) → startup storage reconciliation (override stats from live store). Spans [Server.hs](modules/Simplex/FileTransfer/Server.md), [Server/Store.hs](modules/Simplex/FileTransfer/Server/Store.md), [Server/Env.hs](modules/Simplex/FileTransfer/Server/Env.md), [Server/StoreLog.hs](modules/Simplex/FileTransfer/Server/StoreLog.md) (error-resilient replay, compaction).
|
||||
|
||||
- **XFTP worker architecture**: Five worker types in three categories: rcv (per-router download + local decryption), snd (local prepare/encrypt + per-router upload), del (per-router delete). TMVar-based connection sharing with async retry on temporary errors, permanent error cleanup (put Left + delete from TMap). `withRetryIntervalLimit` caps consecutive retries; exhausted temporary errors silently abandon work cycle (chunk stays pending). `assertAgentForeground` dual check (throw if inactive + wait if backgrounded) gates every chunk operation. Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md), [Client/Agent.hs](modules/Simplex/FileTransfer/Client/Agent.md).
|
||||
- **XFTP worker architecture**: Five worker types in three categories: rcv (per-router data packet download + local decryption), snd (local prepare/encrypt + per-router data packet upload), del (per-router data packet delete). TMVar-based connection sharing with async retry on temporary errors, permanent error cleanup (put Left + delete from TMap). `withRetryIntervalLimit` caps consecutive retries; exhausted temporary errors silently abandon work cycle (chunk stays pending). `assertAgentForeground` dual check (throw if inactive + wait if backgrounded) gates every data packet operation. Spans [Agent.hs](modules/Simplex/FileTransfer/Agent.md), [Client/Agent.hs](modules/Simplex/FileTransfer/Client/Agent.md).
|
||||
|
||||
- **SessionVar protocol client lifecycle**: Protocol client connections (SMP, NTF, XFTP) use a lazy singleton pattern: `getSessVar` atomically checks TMap → `newProtocolClient` fills TMVar on success/failure → `waitForProtocolClient` reads with timeout. Error caching via `persistErrorInterval` prevents connection storms (failed connections cache the error with expiry; callers receive cached error without reconnecting). `removeSessVar` uses monotonic `sessionVarId` compare-and-swap to prevent stale disconnect callbacks from removing newer clients. SMP has additional complexity: `SMPConnectedClient` wraps client with per-connection proxied relay map, `updateClientService` synchronizes service credentials post-connect, disconnect callback moves subscriptions to pending with session-ID matching. XFTP always uses `NRMBackground` timing regardless of caller request. Spans [Session.md](modules/Simplex/Messaging/Session.md), [Agent/Client.md](modules/Simplex/Messaging/Agent/Client.md) (lifecycle, disconnect callbacks, reconnection workers), [Agent.md](modules/Simplex/Messaging/Agent.md) (subscriber loop consuming events).
|
||||
|
||||
|
||||
@@ -4,17 +4,21 @@
|
||||
|
||||
**Source**: [`FileTransfer/Agent.hs`](../../../../src/Simplex/FileTransfer/Agent.hs)
|
||||
|
||||
## Terminology
|
||||
|
||||
The agent splits a **file** into **chunks** determined by the chunking algorithm. Each chunk is stored on an XFTP router as a **data packet** — the router has no concept of files or chunks, only directly addressable data packets. This document uses "chunk" for the agent's internal tracking and "data packet" when referring to what is transferred to/from or stored on routers.
|
||||
|
||||
## Architecture
|
||||
|
||||
The XFTP agent uses five worker types organized in three categories:
|
||||
|
||||
| Worker | Key (router) | Purpose |
|
||||
|--------|-------------|---------|
|
||||
| `xftpRcvWorker` | `Just server` | Download chunks from a specific XFTP router |
|
||||
| `xftpRcvWorker` | `Just server` | Download data packets from a specific XFTP router |
|
||||
| `xftpRcvLocalWorker` | `Nothing` | Decrypt completed downloads locally |
|
||||
| `xftpSndPrepareWorker` | `Nothing` | Encrypt files and create chunks on routers |
|
||||
| `xftpSndWorker` | `Just server` | Upload chunks to a specific XFTP router |
|
||||
| `xftpDelWorker` | `Just server` | Delete chunks from a specific XFTP router |
|
||||
| `xftpSndPrepareWorker` | `Nothing` | Encrypt files and create data packets on routers |
|
||||
| `xftpSndWorker` | `Just server` | Upload data packets to a specific XFTP router |
|
||||
| `xftpDelWorker` | `Just server` | Delete data packets from a specific XFTP router |
|
||||
|
||||
Workers are created on-demand via `getAgentWorker` and keyed by router address. The local workers (keyed by `Nothing`) handle CPU-bound operations that don't require network access.
|
||||
|
||||
@@ -55,7 +59,7 @@ Similarly, `prepareFile` checks `status /= SFSEncrypted` and deletes the partial
|
||||
|
||||
### 8. addRecipients recursive batching
|
||||
|
||||
During upload, `addRecipients` recursively calls itself if a chunk needs more recipients than `xftpMaxRecipientsPerRequest`. Each iteration sends an FADD command for up to `maxRecipients` new recipients, accumulates the results, and recurses until all recipients are registered.
|
||||
During upload, `addRecipients` recursively calls itself if a data packet needs more recipients than `xftpMaxRecipientsPerRequest`. Each iteration sends an FADD command for up to `maxRecipients` new recipients, accumulates the results, and recurses until all recipients are registered.
|
||||
|
||||
### 9. File description generation cross-product
|
||||
|
||||
@@ -71,7 +75,7 @@ During upload, `addRecipients` recursively calls itself if a chunk needs more re
|
||||
|
||||
### 12. Delete workers skip files older than rcvFilesTTL
|
||||
|
||||
`runXFTPDelWorker` uses `rcvFilesTTL` (not a dedicated delete TTL) to filter pending deletions. Files older than this TTL would already be expired on the router, so attempting deletion is pointless. This reuses the receive TTL as a proxy for router-side expiration.
|
||||
`runXFTPDelWorker` uses `rcvFilesTTL` (not a dedicated delete TTL) to filter pending deletions. Data packets older than this TTL would already be expired on the router, so attempting deletion is pointless. This reuses the receive TTL as a proxy for router-side expiration.
|
||||
|
||||
### 13. closeXFTPAgent atomically swaps worker maps
|
||||
|
||||
@@ -83,4 +87,4 @@ During upload, `addRecipients` recursively calls itself if a chunk needs more re
|
||||
|
||||
### 15. Per-router stats tracking
|
||||
|
||||
Every chunk download, upload, and delete operation increments per-router statistics (`downloads`, `uploads`, `deletions`, `downloadAttempts`, `uploadAttempts`, `deleteAttempts`, and error variants). Size-based stats (`downloadsSize`, `uploadsSize`) track throughput in kilobytes.
|
||||
Every data packet download, upload, and delete operation increments per-router statistics (`downloads`, `uploads`, `deletions`, `downloadAttempts`, `uploadAttempts`, `deleteAttempts`, and error variants). Size-based stats (`downloadsSize`, `uploadsSize`) track throughput in kilobytes.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Client
|
||||
|
||||
> XFTP client: connection management, handshake, chunk upload/download with forward secrecy.
|
||||
> XFTP client: connection management, handshake, data packet upload/download with forward secrecy.
|
||||
|
||||
**Source**: [`FileTransfer/Client.hs`](../../../../src/Simplex/FileTransfer/Client.hs)
|
||||
|
||||
@@ -18,19 +18,19 @@
|
||||
|
||||
### 3. Ephemeral DH key pair per download
|
||||
|
||||
`downloadXFTPChunk` generates a fresh X25519 key pair for each chunk download. The public key is sent with the FGET command; the router responds with its own ephemeral key. The derived shared secret encrypts the file data in transit. This provides forward secrecy — compromising a past DH key doesn't decrypt other downloads.
|
||||
`downloadXFTPChunk` generates a fresh X25519 key pair for each data packet download. The public key is sent with the FGET command; the router returns its own ephemeral key. The derived shared secret encrypts the data packet in transit. This provides forward secrecy — compromising a past DH key doesn't decrypt other downloads.
|
||||
|
||||
### 4. Chunk-size-proportional download timeout
|
||||
### 4. Size-proportional download timeout
|
||||
|
||||
`downloadXFTPChunk` calculates the timeout as `baseTimeout + (sizeInKB * perKbTimeout)`, where `baseTimeout` is the base TCP timeout and `perKbTimeout` is a per-kilobyte timeout from the network config. Larger chunks get proportionally more time. This prevents premature timeouts on large chunks over slow connections.
|
||||
`downloadXFTPChunk` calculates the timeout as `baseTimeout + (sizeInKB * perKbTimeout)`, where `baseTimeout` is the base TCP timeout and `perKbTimeout` is a per-kilobyte timeout from the network config. Larger data packets get proportionally more time. This prevents premature timeouts on large data packets over slow connections.
|
||||
|
||||
### 5. prepareChunkSizes threshold algorithm
|
||||
|
||||
`prepareChunkSizes` selects chunk sizes using a 75% threshold: if the remaining payload exceeds 75% of the next larger chunk size, it uses the larger size. Otherwise, it uses the smaller size. `singleChunkSize` returns `Just size` only if the payload fits in a single chunk (used for redirect files which must be single-chunk).
|
||||
`prepareChunkSizes` selects data packet sizes using a 75% threshold: if the remaining payload exceeds 75% of the next larger size, it uses the larger size. Otherwise, it uses the smaller size. `singleChunkSize` returns `Just size` only if the payload fits in a single data packet (used for redirect files which must be single-packet).
|
||||
|
||||
### 6. Upload sends file body after command response
|
||||
### 6. Upload sends data packet after command block
|
||||
|
||||
`uploadXFTPChunk` sends the FPUT command and file body in the same streaming HTTP/2 request: the protocol command block is sent first, followed immediately by the raw file data via `hSendFile`. The router response (`FROk` or error) is received only after both the command and file body have been fully sent. This is a single HTTP/2 round trip, not a two-phase interaction.
|
||||
`uploadXFTPChunk` sends the FPUT command and data packet body in the same streaming HTTP/2 request: the protocol command block is sent first, followed immediately by the raw encrypted data via `hSendFile`. The command result (`FROk` or error) is received only after both the command and data have been fully sent. This is a single HTTP/2 round trip, not a two-phase interaction.
|
||||
|
||||
### 7. Empty corrId as nonce
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Client.Agent
|
||||
|
||||
> XFTP client connection management with TMVar-based sharing, async retry, and connection lifecycle.
|
||||
> XFTP client: router connection management with TMVar-based sharing, async retry, and connection lifecycle.
|
||||
|
||||
**Source**: [`FileTransfer/Client/Agent.hs`](../../../../../src/Simplex/FileTransfer/Client/Agent.hs)
|
||||
|
||||
@@ -24,4 +24,4 @@ On permanent error, `newXFTPClient` puts the `Left error` into the `TMVar` (unbl
|
||||
|
||||
### 5. closeXFTPServerClient removes from TMap
|
||||
|
||||
Closing a router client deletes its entry from the TMap, so the next request will establish a fresh connection. This is called on connection errors during file operations to force reconnection.
|
||||
Closing a router client deletes its entry from the TMap, so the next request will establish a fresh connection. This is called on connection errors during data packet operations to force reconnection.
|
||||
|
||||
@@ -18,9 +18,9 @@
|
||||
|
||||
`receive` tracks a `depth` parameter starting at 1. After following one redirect, `depth` becomes 0. A second redirect throws "Redirect chain too long". This prevents infinite redirect loops from malicious file descriptions.
|
||||
|
||||
### 4. Parallel chunk uploads with router grouping
|
||||
### 4. Parallel data packet uploads with router grouping
|
||||
|
||||
`uploadFile` groups chunks by router via `groupAllOn`, then uses `pooledForConcurrentlyN 16` to process up to 16 router-groups concurrently. Within each group, chunks are uploaded sequentially (`mapM`). Errors from any chunk are collected and the first one is thrown.
|
||||
`uploadFile` groups data packets by router via `groupAllOn`, then uses `pooledForConcurrentlyN 16` to process up to 16 router-groups concurrently. Within each group, data packets are uploaded sequentially (`mapM`). Errors from any upload are collected and the first one is thrown.
|
||||
|
||||
### 5. Random router selection
|
||||
|
||||
@@ -36,8 +36,8 @@
|
||||
|
||||
### 8. File description auto-deletion prompt
|
||||
|
||||
After successful receive or delete, `removeFD` either auto-deletes the file description (if `--yes` flag) or prompts the user. This prevents accidental reuse of one-time file descriptions — each receive consumes the description by ACKing chunks on the router.
|
||||
After successful receive or delete, `removeFD` either auto-deletes the file description (if `--yes` flag) or prompts the user. This prevents accidental reuse of one-time file descriptions — each receive consumes the description by ACKing data packets on the router.
|
||||
|
||||
### 9. Sender description uses first replica's router
|
||||
|
||||
`createSndFileDescription` takes the router from the first replica of each chunk for the sender's `FileChunkReplica`. This reflects the current limitation that each chunk is uploaded to exactly one router — the sender description records that single router.
|
||||
`createSndFileDescription` takes the router from the first replica of each chunk for the sender's `FileChunkReplica`. This reflects the current limitation that each data packet is uploaded to exactly one router — the sender description records that single router.
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
### 2. Fixed-size padding hides actual file size
|
||||
|
||||
The encrypted output is padded to `encSize` (the sum of chunk sizes). Since chunk sizes are fixed powers of 2 (64KB, 256KB, 1MB, 4MB), the encrypted file size reveals only which chunk size bucket the file falls into, not the actual size. The encryption streams data with `LC.sbEncryptChunk` in a loop, pads the remaining space, then manually appends the auth tag via `LC.sbAuth`. This manual streaming approach (rather than using the all-at-once `LC.sbEncryptTailTag`) is necessary because encryption is interleaved with file I/O.
|
||||
The encrypted output is padded to `encSize` (the sum of data packet sizes). Since data packet sizes are fixed powers of 2 (64KB, 256KB, 1MB, 4MB), the encrypted file size reveals only which size bucket the file falls into, not the actual size. The encryption streams data with `LC.sbEncryptChunk` in a loop, pads the remaining space, then manually appends the auth tag via `LC.sbAuth`. This manual streaming approach (rather than using the all-at-once `LC.sbEncryptTailTag`) is necessary because encryption is interleaved with file I/O.
|
||||
|
||||
### 3. Dual decrypt paths: single-chunk vs multi-chunk
|
||||
|
||||
@@ -28,4 +28,4 @@ In the multi-chunk streaming path, if `BA.constEq` detects an auth tag mismatch
|
||||
|
||||
### 5. Streaming encryption uses 64KB blocks
|
||||
|
||||
`encryptFile` reads plaintext in 65536-byte blocks (`LC.sbEncryptChunk`), regardless of the XFTP chunk size. These are encryption blocks within a single continuous stream — not to be confused with XFTP protocol chunks which are much larger (64KB–4MB).
|
||||
`encryptFile` reads plaintext in 65536-byte blocks (`LC.sbEncryptChunk`), regardless of the XFTP data packet size. These are encryption blocks within a single continuous stream — not to be confused with XFTP data packets which are much larger (64KB–4MB).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Description
|
||||
|
||||
> File description: YAML encoding/decoding, validation, URI format, and replica optimization.
|
||||
> File description: YAML encoding/decoding, validation, URI format, and replica optimization. A file description maps a file's chunks to data packets stored on XFTP routers — each chunk corresponds to one data packet, and each data packet may have multiple replicas on different routers.
|
||||
|
||||
**Source**: [`FileTransfer/Description.hs`](../../../../src/Simplex/FileTransfer/Description.hs)
|
||||
|
||||
@@ -24,7 +24,7 @@ The top-level `FileDescription` has a `chunkSize` field. Individual chunk replic
|
||||
|
||||
### 4. YAML encoding groups replicas by router
|
||||
|
||||
`groupReplicasByServer` groups all chunk replicas by their router, producing `FileServerReplica` records. This is the serialization format — replicas are organized by router, not by chunk. The parser (`foldReplicasToChunks`) reverses this grouping back to per-chunk replica lists.
|
||||
`groupReplicasByServer` groups all data packet replicas by their router, producing `FileServerReplica` records. This is the serialization format — replicas are organized by router, not by chunk. The parser (`foldReplicasToChunks`) reverses this grouping back to per-chunk replica lists.
|
||||
|
||||
### 5. FileDescriptionURI uses query-string encoding
|
||||
|
||||
@@ -40,4 +40,4 @@ Two limits exist: `maxFileSize = 1GB` (soft limit, checked by CLI client) and `m
|
||||
|
||||
### 8. Redirect file descriptions
|
||||
|
||||
A `FileDescription` can contain a `redirect` field pointing to another file's metadata (`RedirectFileInfo` with size and digest). The outer description downloads an encrypted YAML file that, once decrypted, yields the actual `FileDescription` for the real file. This adds one level of indirection for privacy — the relay routers hosting the redirect don't know the actual file's routers.
|
||||
A `FileDescription` can contain a `redirect` field pointing to another file's metadata (`RedirectFileInfo` with size and digest). The outer description downloads an encrypted YAML data packet that, once decrypted, yields the actual `FileDescription` for the real file. This adds one level of indirection for privacy — the routers hosting the redirect data packet don't know the actual file's routers.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Protocol
|
||||
|
||||
> XFTP protocol types, commands, responses, and credential verification.
|
||||
> XFTP protocol types, commands, command results, and credential verification.
|
||||
|
||||
**Source**: [`FileTransfer/Protocol.hs`](../../../../src/Simplex/FileTransfer/Protocol.hs)
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
|
||||
This asymmetry means FNEW and PING bypass the standard entity-lookup path entirely — they are handled as separate `XFTPRequest` constructors (`XFTPReqNew`, `XFTPReqPing`).
|
||||
|
||||
### 2. BLOCKED response downgraded to AUTH for old clients
|
||||
### 2. BLOCKED result downgraded to AUTH for old clients
|
||||
|
||||
`encodeProtocol` checks the protocol version: if `v < blockedFilesXFTPVersion`, a `BLOCKED` response is encoded as `AUTH` instead. This prevents old clients that don't understand `BLOCKED` from receiving an unknown error type. The blocking information is silently lost for these clients.
|
||||
`encodeProtocol` checks the protocol version: if `v < blockedFilesXFTPVersion`, a `BLOCKED` result is encoded as `AUTH` instead. This prevents old clients that don't understand `BLOCKED` from receiving an unknown error type. The blocking information is silently lost for these clients.
|
||||
|
||||
### 3. Single-transmission batch enforcement
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Server
|
||||
|
||||
> XFTP router: HTTP/2 request handling, handshake state machine, file operations, and statistics.
|
||||
> XFTP router: HTTP/2 request handling, handshake state machine, data packet operations, and statistics.
|
||||
|
||||
**Source**: [`FileTransfer/Server.hs`](../../../../src/Simplex/FileTransfer/Server.hs)
|
||||
|
||||
@@ -10,8 +10,8 @@ The XFTP router runs several concurrent threads via `raceAny_`:
|
||||
|
||||
| Thread | Purpose |
|
||||
|--------|---------|
|
||||
| `runServer` | HTTP/2 router accepting file transfer requests |
|
||||
| `expireFiles` | Periodic file expiration with throttling |
|
||||
| `runServer` | HTTP/2 router accepting data packet transfer requests |
|
||||
| `expireFiles` | Periodic data packet expiration with throttling |
|
||||
| `logServerStats` | Periodic stats flush to CSV |
|
||||
| `savePrometheusMetrics` | Periodic Prometheus metrics dump |
|
||||
| `runCPServer` | Control port for admin commands |
|
||||
@@ -29,15 +29,15 @@ Web clients can re-send hello (`xftp-web-hello` header) even in `HandshakeSent`
|
||||
|
||||
### 2. Web identity proof via challenge-response
|
||||
|
||||
When a web client sends a hello with a non-empty body, the router parses an `XFTPClientHello` containing a `webChallenge`. The router signs `challenge <> sessionId` with its long-term key and includes the signature in the handshake response. This proves router identity to web clients that cannot verify TLS certificates directly.
|
||||
When a web client sends a hello with a non-empty body, the router parses an `XFTPClientHello` containing a `webChallenge`. The router signs `challenge <> sessionId` with its long-term key and includes the signature in the handshake result. This proves router identity to web clients that cannot verify TLS certificates directly.
|
||||
|
||||
### 3. skipCommitted drains request body on re-upload
|
||||
|
||||
If `receiveServerFile` detects the file is already uploaded (`filePath` TVar is `Just`), it cannot simply ignore the request body — the HTTP/2 client would block waiting for the router to consume it. Instead, `skipCommitted` reads and discards the entire body in `fileBlockSize` increments, returning `FROk` when complete. This makes FPUT idempotent from the client's perspective.
|
||||
If `receiveServerFile` detects the data packet is already uploaded (`filePath` TVar is `Just`), it cannot simply ignore the request body — the HTTP/2 client would block waiting for the router to consume it. Instead, `skipCommitted` reads and discards the entire body in `fileBlockSize` increments, returning `FROk` when complete. This makes FPUT idempotent from the client's perspective.
|
||||
|
||||
### 4. Atomic quota reservation with rollback
|
||||
|
||||
`receiveServerFile` uses `stateTVar` to atomically check and reserve storage quota before receiving the file. If the upload fails (timeout, size mismatch, IO error), the reserved size is subtracted from `usedStorage` and the partial file is deleted on the router. This prevents failed uploads from permanently consuming quota.
|
||||
`receiveServerFile` uses `stateTVar` to atomically check and reserve storage quota before receiving the data packet. If the upload fails (timeout, size mismatch, IO error), the reserved size is subtracted from `usedStorage` and the partial data packet is deleted on the router. This prevents failed uploads from permanently consuming quota.
|
||||
|
||||
### 5. retryAdd generates new IDs on collision
|
||||
|
||||
@@ -45,7 +45,7 @@ If `receiveServerFile` detects the file is already uploaded (`filePath` TVar is
|
||||
|
||||
### 6. Timing attack mitigation on entity lookup
|
||||
|
||||
`verifyXFTPTransmission` calls `dummyVerifyCmd` (imported from SMP router) when a file entity is not found. This equalizes response timing to prevent attackers from distinguishing "entity doesn't exist" from "signature invalid" based on latency.
|
||||
`verifyXFTPTransmission` calls `dummyVerifyCmd` (imported from SMP router) when a data packet entity is not found. This equalizes result timing to prevent attackers from distinguishing "entity doesn't exist" from "signature invalid" based on latency.
|
||||
|
||||
### 7. BLOCKED vs EntityOff distinction
|
||||
|
||||
@@ -56,30 +56,30 @@ When `verifyXFTPTransmission` reads `fileStatus`:
|
||||
|
||||
`EntityOff` is treated identically to missing entities for information-hiding purposes.
|
||||
|
||||
### 8. blockServerFile deletes the physical file
|
||||
### 8. blockServerFile deletes the stored data packet
|
||||
|
||||
Despite the name suggesting it only marks a file as blocked, `blockServerFile` also deletes the physical file from disk via `deleteOrBlockServerFile_`. The `deleted = True` parameter to `blockFile` in the store adjusts `usedStorage`. A blocked file returns `BLOCKED` errors on access but has no data on disk.
|
||||
Despite the name suggesting it only marks a data packet as blocked, `blockServerFile` also deletes the stored data packet from disk via `deleteOrBlockServerFile_`. The `deleted = True` parameter to `blockFile` in the store adjusts `usedStorage`. A blocked data packet returns `BLOCKED` errors on access but has no data on disk.
|
||||
|
||||
### 9. Stats restore overrides counts from live store
|
||||
|
||||
`restoreServerStats` loads stats from the backup file but overrides `_filesCount` and `_filesSize` with values computed from the live file store (TMap size and `usedStorage` TVar). If the backup values differ, warnings are logged. This handles cases where files were expired or deleted while the router was down.
|
||||
`restoreServerStats` loads stats from the backup file but overrides `_filesCount` and `_filesSize` with values computed from the live file store (TMap size and `usedStorage` TVar). If the backup values differ, warnings are logged. This handles cases where data packets were expired or deleted while the router was down.
|
||||
|
||||
### 10. File expiration with configurable throttling
|
||||
### 10. Data packet expiration with configurable throttling
|
||||
|
||||
`expireServerFiles` accepts an optional `itemDelay` (100ms when called from the periodic thread, `Nothing` at router startup). Between each file check, `threadDelay itemDelay` prevents expiration from monopolizing IO. At startup, files are expired without delay to clean up quickly.
|
||||
`expireServerFiles` accepts an optional `itemDelay` (100ms when called from the periodic thread, `Nothing` at router startup). Between each data packet check, `threadDelay itemDelay` prevents expiration from monopolizing IO. At startup, data packets are expired without delay to clean up quickly.
|
||||
|
||||
### 11. Stats log aligns to wall-clock midnight
|
||||
|
||||
`logServerStats` computes an `initialDelay` to align the first stats flush to `logStatsStartTime` (default 0 = midnight UTC). If the target time already passed today, it adds 86400 seconds for the next day. Subsequent flushes use exact `logInterval` cadence.
|
||||
|
||||
### 12. Physical file deleted before store cleanup
|
||||
### 12. Stored data packet deleted before store cleanup
|
||||
|
||||
`deleteOrBlockServerFile_` removes the physical file first, then runs the STM store action. If the process crashes between these two operations, the store will reference a file that no longer exists on disk. The next access would return `AUTH` (file not found on disk), and eventual expiration would clean the store entry.
|
||||
`deleteOrBlockServerFile_` removes the stored data packet first, then runs the STM store action. If the process crashes between these two operations, the store will reference a data packet that no longer exists on disk. The next access would return `AUTH` (data packet not found on disk), and eventual expiration would clean the store entry.
|
||||
|
||||
### 13. SNI-dependent CORS and web serving
|
||||
|
||||
CORS headers require both `sniUsed = True` and `addCORSHeaders = True` in the transport config. Static web page serving is enabled when `sniUsed = True`. Non-SNI connections (direct TLS without hostname) skip both CORS and web serving. This separates the web-facing and protocol-facing behaviors of the same router port.
|
||||
|
||||
### 14. Control port file operations use recipient index
|
||||
### 14. Control port data packet operations use recipient index
|
||||
|
||||
`CPDelete` and `CPBlock` commands look up files via `getFile fs SFRecipient fileId`, meaning the control port takes a recipient ID, not a sender ID. This is the ID visible to recipients and contained in file descriptions.
|
||||
`CPDelete` and `CPBlock` commands look up data packets via `getFile fs SFRecipient fileId`, meaning the control port takes a recipient ID, not a sender ID. This is the ID visible to recipients and contained in data packet descriptions.
|
||||
|
||||
@@ -8,17 +8,17 @@
|
||||
|
||||
### 1. Startup storage accounting with quota warning
|
||||
|
||||
`newXFTPServerEnv` computes `usedStorage` by summing file sizes from the in-memory store at startup. If the computed usage exceeds the configured `fileSizeQuota`, a warning is logged but the router still starts. This allows the router to come up even if it's over quota (e.g., after a quota reduction), relying on expiration to reclaim space.
|
||||
`newXFTPServerEnv` computes `usedStorage` by summing data packet sizes from the in-memory store at startup. If the computed usage exceeds the configured `fileSizeQuota`, a warning is logged but the router still starts. This allows the router to come up even if it's over quota (e.g., after a quota reduction), relying on expiration to reclaim space.
|
||||
|
||||
### 2. XFTPRequest ADT separates new files from commands
|
||||
### 2. XFTPRequest ADT separates new data packets from commands
|
||||
|
||||
`XFTPRequest` has three constructors:
|
||||
- `XFTPReqNew`: file creation (carries `FileInfo`, recipient keys, optional basic auth)
|
||||
- `XFTPReqCmd`: command on an existing file (carries file ID, `FileRec`, and the command)
|
||||
- `XFTPReqNew`: data packet creation (carries `FileInfo`, recipient keys, optional basic auth)
|
||||
- `XFTPReqCmd`: command on an existing data packet (carries file ID, `FileRec`, and the command)
|
||||
- `XFTPReqPing`: health check
|
||||
|
||||
This separation occurs after credential verification in `Server.hs`. `XFTPReqNew` bypasses entity lookup entirely since the file doesn't exist yet.
|
||||
This separation occurs after credential verification in `Server.hs`. `XFTPReqNew` bypasses entity lookup entirely since the data packet doesn't exist yet.
|
||||
|
||||
### 3. fileTimeout for upload deadline
|
||||
|
||||
`fileTimeout` in `XFTPServerConfig` sets the maximum time allowed for a single file upload (FPUT). The router wraps the receive operation in `timeout fileTimeout`. Default is 5 minutes (for 4MB chunks). This prevents slow or stalled uploads from holding router resources indefinitely.
|
||||
`fileTimeout` in `XFTPServerConfig` sets the maximum time allowed for a single data packet upload (FPUT). The router wraps the receive operation in `timeout fileTimeout`. Default is 5 minutes (for 4MB chunks). This prevents slow or stalled uploads from holding router resources indefinitely.
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
| Constant | Value | Purpose |
|
||||
|----------|-------|---------|
|
||||
| `fileIdSize` | 16 bytes | Random file/recipient ID length |
|
||||
| `fileIdSize` | 16 bytes | Random data packet/recipient ID length |
|
||||
| `fileTimeout` | 5 minutes | Maximum upload duration per chunk |
|
||||
| `logStatsInterval` | 86400s (daily) | Stats CSV flush interval |
|
||||
| `logStatsStartTime` | 0 (midnight UTC) | First stats flush time-of-day |
|
||||
|
||||
@@ -16,4 +16,4 @@ The `strP` parser uses `opt` for newer fields, defaulting missing fields to 0. T
|
||||
|
||||
### 3. PeriodStats for download tracking
|
||||
|
||||
`filesDownloaded` uses `PeriodStats` (not a simple `IORef Int`) to track unique file downloads over time periods (day/week/month). This enables the CSV stats log to report distinct files downloaded per period, not just total download count.
|
||||
`filesDownloaded` uses `PeriodStats` (not a simple `IORef Int`) to track unique data packet downloads over time periods (day/week/month). This enables the CSV stats log to report distinct data packets downloaded per period, not just total download count.
|
||||
|
||||
@@ -16,24 +16,24 @@ The file store maintains two indices: `files :: TMap SenderId FileRec` (by sende
|
||||
|
||||
### 3. Storage accounting on upload completion
|
||||
|
||||
`setFilePath` adds the file size to `usedStorage` and records the file path in the `filePath` TVar. However, during normal FPUT handling, `Server.hs` does NOT call `setFilePath` — it directly writes `filePath` via `writeTVar`. The quota reservation in `Server.hs` (`stateTVar` on `usedStorage`) is the sole `usedStorage` increment during upload. `setFilePath` IS called during store log replay (`StoreLog.hs`), where it increments `usedStorage`; `newXFTPServerEnv` then overwrites with the correct value computed from the live store.
|
||||
`setFilePath` adds the data packet size to `usedStorage` and records the file path in the `filePath` TVar. However, during normal FPUT handling, `Server.hs` does NOT call `setFilePath` — it directly writes `filePath` via `writeTVar`. The quota reservation in `Server.hs` (`stateTVar` on `usedStorage`) is the sole `usedStorage` increment during upload. `setFilePath` IS called during store log replay (`StoreLog.hs`), where it increments `usedStorage`; `newXFTPServerEnv` then overwrites with the correct value computed from the live store.
|
||||
|
||||
### 4. deleteFile removes all recipients atomically
|
||||
|
||||
`deleteFile` atomically removes the sender entry from `files`, all recipient entries from the global `recipients` TMap, and unconditionally subtracts the file size from `usedStorage` (regardless of whether the file was actually uploaded). The entire operation runs in a single STM transaction.
|
||||
`deleteFile` atomically removes the sender entry from `files`, all recipient entries from the global `recipients` TMap, and unconditionally subtracts the data packet size from `usedStorage` (regardless of whether the data packet was actually uploaded). The entire operation runs in a single STM transaction.
|
||||
|
||||
### 5. RoundedSystemTime for privacy-preserving expiration
|
||||
|
||||
File timestamps use `RoundedFileTime` which is `RoundedSystemTime 3600` — system time rounded to 1-hour precision. This means files created within the same hour have identical timestamps. An observer with access to the store cannot determine exact file creation times, only the hour.
|
||||
Data packet timestamps use `RoundedFileTime` which is `RoundedSystemTime 3600` — system time rounded to 1-hour precision. This means data packets created within the same hour have identical timestamps. An observer with access to the store cannot determine exact data packet creation times, only the hour.
|
||||
|
||||
### 6. expiredFilePath returns path only if expired
|
||||
|
||||
`expiredFilePath` returns `STM (Maybe (Maybe FilePath))`. The outer `Maybe` is `Nothing` when the file doesn't exist or isn't expired; the inner `Maybe` is the file path (present only if the file was uploaded). The expiration check adds `fileTimePrecision` (one hour) to the creation timestamp before comparing, providing a grace period. The caller uses the inner path to decide whether to also delete the physical file.
|
||||
`expiredFilePath` returns `STM (Maybe (Maybe FilePath))`. The outer `Maybe` is `Nothing` when the data packet doesn't exist or isn't expired; the inner `Maybe` is the file path (present only if the data packet was uploaded). The expiration check adds `fileTimePrecision` (one hour) to the creation timestamp before comparing, providing a grace period. The caller uses the inner path to decide whether to also delete the stored data packet.
|
||||
|
||||
### 7. ackFile removes single recipient
|
||||
|
||||
`ackFile` removes a specific recipient from both the global `recipients` TMap and the per-file `recipientIds` Set. Unlike `deleteFile` which removes the entire file, `ackFile` only removes one recipient's access. The file and other recipients remain intact.
|
||||
`ackFile` removes a specific recipient from both the global `recipients` TMap and the per-file `recipientIds` Set. Unlike `deleteFile` which removes the entire data packet, `ackFile` only removes one recipient's access. The data packet and other recipients remain intact.
|
||||
|
||||
### 8. blockFile conditional storage adjustment
|
||||
|
||||
`blockFile` takes a `deleted :: Bool` parameter. When `True` (file blocked with physical deletion), it subtracts the file size from `usedStorage`. When `False` (block without deletion), storage is unchanged. This allows blocking without physical deletion for audit purposes. Currently, both the router's `blockServerFile` and the store log replay path pass `True`.
|
||||
`blockFile` takes a `deleted :: Bool` parameter. When `True` (data packet blocked with physical deletion), it subtracts the data packet size from `usedStorage`. When `False` (block without deletion), storage is unchanged. This allows blocking without physical deletion for audit purposes. Currently, both the router's `blockServerFile` and the store log replay path pass `True`.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Server.StoreLog
|
||||
|
||||
> Append-only store log for XFTP router file operations with error-resilient replay and compaction.
|
||||
> Append-only store log for XFTP router data packet operations with error-resilient replay and compaction.
|
||||
|
||||
**Source**: [`FileTransfer/Server/StoreLog.hs`](../../../../../src/Simplex/FileTransfer/Server/StoreLog.hs)
|
||||
|
||||
@@ -24,10 +24,10 @@
|
||||
|
||||
### 5. Log entry types track operation lifecycle
|
||||
|
||||
Six log entry types capture the complete file lifecycle:
|
||||
- `AddFile`: file creation with sender ID, file info, timestamp, and status
|
||||
Six log entry types capture the complete data packet lifecycle:
|
||||
- `AddFile`: data packet creation with sender ID, file info, timestamp, and status
|
||||
- `AddRecipients`: recipient registration (batched as `NonEmpty FileRecipient`) with sender ID association
|
||||
- `PutFile`: upload completion with file path
|
||||
- `DeleteFile`: file deletion by sender ID
|
||||
- `DeleteFile`: data packet deletion by sender ID
|
||||
- `AckFile`: single recipient acknowledgment
|
||||
- `BlockFile`: file blocking with blocking info
|
||||
- `BlockFile`: data packet blocking with blocking info
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.FileTransfer.Types
|
||||
|
||||
> Agent-side file transfer types: receive/send file records, status state machines, chunk/replica structures.
|
||||
> Agent-side file transfer types: receive/send file records, status state machines, and chunk/replica structures. Chunks are the agent's view of file pieces; each chunk maps to a data packet on an XFTP router.
|
||||
|
||||
**Source**: [`FileTransfer/Types.hs`](../../../../src/Simplex/FileTransfer/Types.hs)
|
||||
|
||||
@@ -24,4 +24,4 @@
|
||||
|
||||
### 5. authTagSize = 16 bytes
|
||||
|
||||
`authTagSize` is defined as `fromIntegral C.authTagSize` (16 bytes). This is the AES-GCM authentication tag appended to the encrypted file stream. It is included in the payload size calculation (`payloadSize = fileSize' + fileSizeLen + authTagSize`), which is then passed to `prepareChunkSizes` to determine chunk allocation.
|
||||
`authTagSize` is defined as `fromIntegral C.authTagSize` (16 bytes). This is the AES-GCM authentication tag appended to the encrypted file stream. It is included in the payload size calculation (`payloadSize = fileSize' + fileSizeLen + authTagSize`), which is then passed to `prepareChunkSizes` to determine data packet allocation.
|
||||
|
||||
@@ -38,9 +38,9 @@ The subscriber thread reads batches from `msgQ` (filled by SMP protocol clients)
|
||||
|
||||
**Batch UP notification accumulation.** Successful subscription confirmations (`processSubOk`) append to a shared `upConnIds` TVar across the batch. A single `UP` event is emitted after all transmissions are processed, not per-transmission. Similarly, `serviceRQs` accumulates service-associated receive queues for batch processing via `processRcvServiceAssocs`.
|
||||
|
||||
**Double validation for subscription results.** `isPendingSub` checks two conditions atomically: the queue must be in the pending map AND the client session must still be active (`activeClientSession`). If either fails, the result is counted as ignored (statistics only). This handles the race where a subscription response arrives after reconnection.
|
||||
**Double validation for subscription results.** `isPendingSub` checks two conditions atomically: the queue must be in the pending map AND the client session must still be active (`activeClientSession`). If either fails, the result is counted as ignored (statistics only). This handles the race where a subscription result arrives after reconnection.
|
||||
|
||||
**SUB response piggybacking MSG.** When a SUB response arrives as `Right msg@SMP.MSG {}`, the connection is marked UP (via `processSubOk`) AND the MSG is processed. The UP notification happens even if the MSG processing fails — the connection is up regardless.
|
||||
**SUB result piggybacking MSG.** When a SUB result arrives as `Right msg@SMP.MSG {}`, the connection is marked UP (via `processSubOk`) AND the MSG is processed. The UP notification happens even if the MSG processing fails — the connection is up regardless.
|
||||
|
||||
**subQ overflow to pendingMsgs.** `processSMP` writes events to `subQ` (bounded TBQueue) but when full, events go into a `pendingMsgs` TVar. After processing, pending messages are drained in reverse order (LIFO). This prevents the message processing thread from blocking on a full queue, which would stall the entire SMP client.
|
||||
|
||||
|
||||
@@ -48,7 +48,7 @@ This is the mechanism for time-scheduled subscription health checks.
|
||||
|
||||
When the notification router returns `AUTH` for a subscription check, the subscription is not simply marked as failed — it is fully recreated from scratch by resetting to `NSASMP NSASmpKey` state. This handles the case where the notification router has lost its subscription state (restart, data loss). The SMP worker is kicked to re-establish notifier credentials.
|
||||
|
||||
Successful check responses with statuses not in `subscribeNtfStatuses` also trigger recreation via `recreateNtfSub`.
|
||||
Successful check results with statuses not in `subscribeNtfStatuses` also trigger recreation via `recreateNtfSub`.
|
||||
|
||||
### 5. deleteToken two-phase with restart survival
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Service subscriptions (aggregate, router-managed) and queue subscriptions (indiv
|
||||
|
||||
The central invariant: a subscription is only active if it was confirmed on the *current* TLS session. Every function that promotes subscriptions to active (`addActiveSub'`, `batchAddActiveSubs`, `setActiveServiceSub`) checks `Just sessId == sessId'` (stored session ID). On mismatch, the subscription goes to pending instead — silently, with no error.
|
||||
|
||||
This means subscription RPCs that succeed but return after a reconnect are safely caught: the response carries the old session ID, which won't match the new one stored by `setSessionId`.
|
||||
This means subscription RPCs that succeed but return after a reconnect are safely caught: the result carries the old session ID, which won't match the new one stored by `setSessionId`.
|
||||
|
||||
## setSessionId — silent demotion on reconnect
|
||||
|
||||
|
||||
@@ -8,37 +8,37 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This module implements the client side of the `Protocol` typeclass — connecting to SMP routers, sending commands, receiving responses, and managing connection lifecycle. It is generic over `Protocol v err msg`, instantiated for SMP as `SMPClient` (= `ProtocolClient SMPVersion ErrorType BrokerMsg`). The SMP proxy protocol (PRXY/PFWD/RFWD) is also implemented here.
|
||||
This module implements the client side of the `Protocol` typeclass — connecting to SMP routers, sending commands, receiving command results, and managing connection lifecycle. It is generic over `Protocol v err msg`, instantiated for SMP as `SMPClient` (= `ProtocolClient SMPVersion ErrorType BrokerMsg`). The SMP proxy protocol (PRXY/PFWD/RFWD) is also implemented here.
|
||||
|
||||
## Four concurrent threads — teardown semantics
|
||||
|
||||
`getProtocolClient` launches four threads via `raceAny_`:
|
||||
- `send`: reads from `sndQ` (TBQueue) and writes to TLS
|
||||
- `receive`: reads from TLS and writes to `rcvQ` (TBQueue), updates `lastReceived`
|
||||
- `process`: reads from `rcvQ` and dispatches to response vars or `msgQ`
|
||||
- `process`: reads from `rcvQ` and dispatches to result vars or `msgQ`
|
||||
- `monitor`: periodic ping loop (only when `smpPingInterval > 0`)
|
||||
|
||||
When ANY thread exits (normally or exceptionally), `raceAny_` cancels all others. `E.finally` ensures the `disconnected` callback always fires. Implication: a single stuck thread (e.g., TLS read blocked on a half-open connection) keeps the entire client alive until `monitor` drops it. There is no per-thread health check — liveness depends entirely on the monitor's timeout logic.
|
||||
|
||||
## Request lifecycle and leak risk
|
||||
|
||||
`mkRequest` inserts a `Request` into `sentCommands` TMap BEFORE the transmission is written to TLS. If the TLS write fails silently or the connection drops before the response, the entry remains in `sentCommands` until the monitor's timeout counter exceeds `maxCnt` and drops the entire client. There is no per-request cleanup on send failure — individual request entries are only removed by `processMsg` (on response) or by `getResponse` timeout (which sets `pending = False` but doesn't remove the entry).
|
||||
`mkRequest` inserts a `Request` into `sentCommands` TMap BEFORE the transmission is written to TLS. If the TLS write fails silently or the connection drops before the result arrives, the entry remains in `sentCommands` until the monitor's timeout counter exceeds `maxCnt` and drops the entire client. There is no per-request cleanup on send failure — individual request entries are only removed by `processMsg` (on result) or by `getResponse` timeout (which sets `pending = False` but doesn't remove the entry).
|
||||
|
||||
## getResponse — pending flag race contract
|
||||
|
||||
This is the core concurrency contract between timeout and response processing:
|
||||
This is the core concurrency contract between timeout and result processing:
|
||||
|
||||
1. `getResponse` waits with `timeout` for `takeTMVar responseVar`
|
||||
2. Regardless of result, atomically sets `pending = False` and tries `tryTakeTMVar` again (see comment on `getResponse`)
|
||||
3. In `processMsg`, when a response arrives for a request where `pending` is already `False` (timeout won), `wasPending` is `False` and the response is forwarded to `msgQ` as `STResponse` rather than discarded
|
||||
3. In `processMsg`, when a result arrives for a request where `pending` is already `False` (timeout won), `wasPending` is `False` and the result is forwarded to `msgQ` as `STResponse` rather than discarded
|
||||
|
||||
The double-check pattern (`swapTVar pending False` + `tryTakeTMVar`) handles the race window where a response arrives between timeout firing and `pending` being set to `False`. Without this, responses arriving in that gap would be silently lost.
|
||||
The double-check pattern (`swapTVar pending False` + `tryTakeTMVar`) handles the race window where a result arrives between timeout firing and `pending` being set to `False`. Without this, results arriving in that gap would be silently lost.
|
||||
|
||||
`timeoutErrorCount` is reset to 0 in three places: in `getResponse` when a response arrives, in `receive` on every TLS read, and the monitor uses this count to decide when to drop the connection.
|
||||
`timeoutErrorCount` is reset to 0 in three places: in `getResponse` when a result arrives, in `receive` on every TLS read, and the monitor uses this count to decide when to drop the connection.
|
||||
|
||||
## processMsg — router events vs expired responses
|
||||
## processMsg — router events vs expired results
|
||||
|
||||
When `corrId` is empty, the message is an `STEvent` (router-initiated). When non-empty and the request was already expired (`wasPending` is `False`), the response becomes `STResponse` — not discarded, but forwarded to `msgQ` with the original command context. Entity ID mismatch is `STUnexpectedError`.
|
||||
When `corrId` is empty, the message is an `STEvent` (router-initiated). When non-empty and the request was already expired (`wasPending` is `False`), the result becomes `STResponse` — not discarded, but forwarded to `msgQ` with the original command context. Entity ID mismatch is `STUnexpectedError`.
|
||||
|
||||
## nonBlockingWriteTBQueue — fork on full
|
||||
|
||||
@@ -46,13 +46,13 @@ If `tryWriteTBQueue` returns `False`, a new thread is forked for the blocking wr
|
||||
|
||||
## Batch commands do not expire
|
||||
|
||||
See comment on `sendBatch`. Batched commands are written with `Nothing` as the request parameter — the send thread skips the `pending` flag check. Individual commands use `Just r` and the send thread checks `pending` after dequeue. The coupling: if the router stops responding, batched commands can block the send queue indefinitely since they have no timeout-based expiry.
|
||||
See comment on `sendBatch`. Batched commands are written with `Nothing` as the request parameter — the send thread skips the `pending` flag check. Individual commands use `Just r` and the send thread checks `pending` after dequeue. The coupling: if the router stops returning results, batched commands can block the send queue indefinitely since they have no timeout-based expiry.
|
||||
|
||||
## monitor — quasi-periodic adaptive ping
|
||||
|
||||
The ping loop sleeps for `smpPingInterval`, then checks elapsed time since `lastReceived`. If significant time remains in the interval (> 1 second), it re-sleeps for just the remaining time rather than sending a ping. This means ping frequency adapts to actual receive activity — frequent receives suppress pings.
|
||||
|
||||
Pings are only sent when `sendPings` is `True`, set by `enablePings` (called from `subscribeSMPQueue`, `subscribeSMPQueues`, `subscribeSMPQueueNotifications`, `subscribeSMPQueuesNtfs`, `subscribeService`). The client drops the connection when `maxCnt` commands have timed out in sequence AND at least `recoverWindow` (15 minutes) has passed since the last received response.
|
||||
Pings are only sent when `sendPings` is `True`, set by `enablePings` (called from `subscribeSMPQueue`, `subscribeSMPQueues`, `subscribeSMPQueueNotifications`, `subscribeSMPQueuesNtfs`, `subscribeService`). The client drops the connection when `maxCnt` commands have timed out in sequence AND at least `recoverWindow` (15 minutes) has passed since the last received result.
|
||||
|
||||
## clientCorrId — dual-purpose random values
|
||||
|
||||
@@ -68,7 +68,7 @@ See comment above `proxySMPCommand` for the 9 error scenarios (0-9) mapping each
|
||||
|
||||
## forwardSMPTransmission — proxy-side forwarding
|
||||
|
||||
Used by the proxy router to forward `RFWD` to the destination relay. Uses `cbEncryptNoPad`/`cbDecryptNoPad` (no padding) with the session secret from the proxy-relay connection. Response nonce is `reverseNonce` of the request nonce.
|
||||
Used by the proxy router to forward `RFWD` to the destination relay. Uses `cbEncryptNoPad`/`cbDecryptNoPad` (no padding) with the session secret from the proxy-relay connection. Result nonce is `reverseNonce` of the request nonce.
|
||||
|
||||
## authTransmission — dual auth with service signature
|
||||
|
||||
@@ -82,4 +82,4 @@ The service signature is only added when the entity authenticator is non-empty.
|
||||
|
||||
## writeSMPMessage — router-side event injection
|
||||
|
||||
`writeSMPMessage` writes directly to `msgQ` as `STEvent`, bypassing the entire command/response pipeline. This is used by the router to inject MSG events into the subscription response path.
|
||||
`writeSMPMessage` writes directly to `msgQ` as `STEvent`, bypassing the entire command/result pipeline. This is used by the router to inject MSG events into the subscription result path.
|
||||
|
||||
@@ -45,9 +45,9 @@ When `connectClient` calls `newSMPClient` and it fails, the error is stored with
|
||||
|
||||
Both `smpSubscribeQueues` and `smpSubscribeService` validate `activeClientSession` AFTER the subscription RPC completes, before committing results to state. If the session changed during the RPC (client reconnected), results are discarded and reconnection is triggered. This is optimistic execution with post-hoc validation — the RPC may succeed but its results are thrown away if the session is stale.
|
||||
|
||||
## groupSub — subscription response classification
|
||||
## groupSub — subscription result classification
|
||||
|
||||
Each queue response is classified by a `foldr` over the (subs, responses) zip:
|
||||
Each queue result is classified by a `foldr` over the (subs, results) zip:
|
||||
|
||||
- **Success with matching serviceId**: counted as service-subscribed (`sQs` list)
|
||||
- **Success without matching serviceId**: counted as queue-only (`qOks` list with SessionId and key)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.Messaging.Notifications.Protocol
|
||||
|
||||
> NTF protocol entities, commands, responses, and wire encoding for the notification system.
|
||||
> NTF protocol entities, commands, command results, and wire encoding for the notification system.
|
||||
|
||||
**Source**: [`Notifications/Protocol.hs`](../../../../../src/Simplex/Messaging/Notifications/Protocol.hs)
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
| PING | No | Must be empty |
|
||||
| All others | Yes | Must be present |
|
||||
|
||||
For responses, the rule inverts: `NRTknId`, `NRSubId`, and `NRPong` must NOT have entity IDs (they are returned before/without entity context), while `NRErr` optionally has one (errors can occur with or without entity context).
|
||||
For command results, the rule inverts: `NRTknId`, `NRSubId`, and `NRPong` must NOT have entity IDs (they are returned before/without entity context), while `NRErr` optionally has one (errors can occur with or without entity context).
|
||||
|
||||
### 2. PNMessageData semicolon separator
|
||||
|
||||
@@ -24,7 +24,7 @@ For responses, the rule inverts: `NRTknId`, `NRSubId`, and `NRPong` must NOT hav
|
||||
|
||||
### 3. NTInvalid reason is version-gated
|
||||
|
||||
When encoding `NRTkn` responses, the `NTInvalid` reason is only included if the negotiated protocol version is >= `invalidReasonNTFVersion` (v3). Older clients receive `NTInvalid Nothing`. This prevents parse failures on clients that don't understand the reason field.
|
||||
When encoding `NRTkn` results, the `NTInvalid` reason is only included if the negotiated protocol version is >= `invalidReasonNTFVersion` (v3). Older clients receive `NTInvalid Nothing`. This prevents parse failures on clients that don't understand the reason field.
|
||||
|
||||
### 4. subscribeNtfStatuses migration invariant
|
||||
|
||||
@@ -46,7 +46,7 @@ Token status `NTInvalid` allows subscription commands (SNEW, SCHK, SDEL), which
|
||||
|
||||
Both `smpP` and `strP` for `SMPQueueNtf` apply `updateSMPServerHosts` to the parsed SMP server. This normalizes router host addresses on deserialization, ensuring consistent comparison even if the on-wire format uses different host representations.
|
||||
|
||||
### 9. NRTknId response tag comment
|
||||
### 9. NRTknId result tag comment
|
||||
|
||||
The `NRTknId_` tag encodes as `"IDTKN"` with a source comment: "it should be 'TID', 'SID'". This indicates a naming inconsistency that was preserved for backward compatibility — the tag names don't follow the pattern of other NTF protocol tags.
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ Each client connection spawns `receive`, `send`, and `client` threads via `raceA
|
||||
|
||||
### 1. Timing attack mitigation on entity lookup
|
||||
|
||||
When `verifyNtfTransmission` encounters an AUTH error (entity not found), it calls `dummyVerifyCmd` to equalize response timing before returning the error. This prevents attackers from distinguishing "entity doesn't exist" from "signature invalid" based on response latency.
|
||||
When `verifyNtfTransmission` encounters an AUTH error (entity not found), it calls `dummyVerifyCmd` to equalize result timing before returning the error. This prevents attackers from distinguishing "entity doesn't exist" from "signature invalid" based on result latency.
|
||||
|
||||
### 2. TNEW idempotent re-registration
|
||||
|
||||
@@ -74,9 +74,9 @@ Cron notification interval has a hard minimum of 20 minutes. `TCRN 0` disables c
|
||||
|
||||
`resubscribe` uses `mapConcurrently` to resubscribe to all known SMP routers in parallel. Within each router, subscriptions are paginated via `subscribeLoop` using cursor-based pagination (`afterSubId_`).
|
||||
|
||||
### 11. receive separates error responses from commands
|
||||
### 11. receive separates error results from commands
|
||||
|
||||
The `receive` function processes incoming transmissions and partitions results: malformed/unauthorized requests are written directly to `sndQ` as error responses, while valid commands go to `rcvQ` for processing. This ensures protocol errors get immediate responses without competing for the command processing queue.
|
||||
The `receive` function processes incoming transmissions and partitions results: malformed/unauthorized requests are written directly to `sndQ` as error results, while valid commands go to `rcvQ` for processing. This ensures protocol errors get immediate results without competing for the command processing queue.
|
||||
|
||||
### 12. Maintenance mode saves state then exits immediately
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ Two feature gates exist in the NTF protocol:
|
||||
| Version | Feature | Effect |
|
||||
|---------|---------|--------|
|
||||
| v2 (`authBatchCmdsNTFVersion`) | Auth key exchange + batching | `authPubKey` sent in handshake, `implySessId` and `batch` enabled |
|
||||
| v3 (`invalidReasonNTFVersion`) | Token invalid reasons | `NTInvalid` responses include the reason enum |
|
||||
| v3 (`invalidReasonNTFVersion`) | Token invalid reasons | `NTInvalid` results include the reason enum |
|
||||
|
||||
Pre-v2 connections have no command encryption or batching — commands are sent in plaintext within TLS.
|
||||
|
||||
@@ -27,7 +27,7 @@ Pre-v2 connections have no command encryption or batching — commands are sent
|
||||
|
||||
### 4. Block size
|
||||
|
||||
NTF uses a 512-byte block size (`ntfBlockSize`), significantly smaller than SMP. This is sufficient because NTF protocol commands (TNEW, SNEW, TCHK, etc.) and their responses are short. `PNMessageData` (which contains encrypted message metadata) is not sent over the NTF transport — it is delivered via APNS push notifications.
|
||||
NTF uses a 512-byte block size (`ntfBlockSize`), significantly smaller than SMP. This is sufficient because NTF protocol commands (TNEW, SNEW, TCHK, etc.) and their results are short. `PNMessageData` (which contains encrypted message metadata) is not sent over the NTF transport — it is delivered via APNS push notifications.
|
||||
|
||||
### 5. Initial THandle has version 0
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.Messaging.Protocol
|
||||
|
||||
> SMP protocol types, commands, responses, encoding/decoding, and transport functions.
|
||||
> SMP protocol types, commands, command results, encoding/decoding, and transport functions.
|
||||
|
||||
**Source**: [`Protocol.hs`](../../../../src/Simplex/Messaging/Protocol.hs)
|
||||
|
||||
@@ -65,4 +65,4 @@ The `NETWORK` variant of `BrokerErrorType` encodes as just `"NETWORK"` (detail d
|
||||
|
||||
## SUBS/NSUBS — asymmetric defaulting
|
||||
|
||||
When the router parses `SUBS`/`NSUBS` from a client using a version older than `rcvServiceSMPVersion`, both count and hash default (`-1` and `mempty`). For the response side (`SOKS`/`ENDS` via `serviceRespP`), count is still parsed from the wire — only hash defaults to `mempty`. This asymmetry means command-side and response-side parsing have different fallback behavior for the same version boundary.
|
||||
When the router parses `SUBS`/`NSUBS` from a client using a version older than `rcvServiceSMPVersion`, both count and hash default (`-1` and `mempty`). For the result side (`SOKS`/`ENDS` via `serviceRespP`), count is still parsed from the wire — only hash defaults to `mempty`. This asymmetry means command-side and result-side parsing have different fallback behavior for the same version boundary.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Simplex.Messaging.Protocol.Types
|
||||
|
||||
> Client notice type with optional TTL, used in BLOCKED error responses.
|
||||
> Client notice type with optional TTL, used in BLOCKED error results.
|
||||
|
||||
**Source**: [`Protocol/Types.hs`](../../../../../src/Simplex/Messaging/Protocol/Types.hs)
|
||||
|
||||
|
||||
@@ -59,7 +59,7 @@ Stats classification: exactly one of `srvSubOk`/`srvSubMore`/`srvSubFewer`/`srvS
|
||||
|
||||
See comment on `processForwardedCommand`. Only single forwarded transmissions are allowed — batches are rejected with `BLOCK`. The synthetic `THandleAuth` has `peerClientService = Nothing`, preventing forwarded clients from claiming service identity. Only SEND, SKEY, LKEY, and LGET are allowed through `rejectOrVerify`.
|
||||
|
||||
Double encryption: response is encrypted first to the client (with `C.cbEncrypt` using `reverseNonce clientNonce`), then wrapped and encrypted to the proxy (with `C.cbEncryptNoPad` using `reverseNonce proxyNonce`). Using reversed nonces ensures request and response directions use distinct nonces.
|
||||
Double encryption: the result is encrypted first to the client (with `C.cbEncrypt` using `reverseNonce clientNonce`), then wrapped and encrypted to the proxy (with `C.cbEncryptNoPad` using `reverseNonce proxyNonce`). Using reversed nonces ensures command and result directions use distinct nonces.
|
||||
|
||||
## Proxy concurrency limiter
|
||||
|
||||
@@ -73,13 +73,13 @@ See `wait`/`signal` around `forkProxiedCmd`. `procThreads` TVar implements a cou
|
||||
|
||||
See `withSubscribed`. When a service client unsubscribes between the TVar read and the flush, `throwSTM (userError "service unsubscribed")` aborts the STM transaction. This is caught by `tryAny` and logged as "cancelled" — it's a successful path, not an error. The `flushSubscribedNtfs` function also cancels via `throwSTM` if the client is no longer current or sndQ is full.
|
||||
|
||||
## Batch subscription responses — SOK grouped with MSG
|
||||
## Batch subscription results — SOK grouped with MSG
|
||||
|
||||
See comment on `processSubBatch`. When batched SUB commands produce SOK responses plus messages, the first message is appended to the SOK batch (up to 4 SOKs per block) in a single transmission. Remaining messages go to `msgQ` for separate delivery. This ensures the client receives at least one message quickly with its subscription acknowledgments.
|
||||
See comment on `processSubBatch`. When batched SUB commands produce SOK results plus messages, the first message is appended to the SOK batch (up to 4 SOKs per block) in a single transmission. Remaining messages go to `msgQ` for separate delivery. This ensures the client receives at least one message quickly with its subscription acknowledgments.
|
||||
|
||||
## send thread — MVar fair lock
|
||||
|
||||
The TLS handle is wrapped in an `MVar` (`newMVar h`). Both `send` (command responses from `sndQ`) and `sendMsg` (messages from `msgQ`) acquire this lock via `withMVar`. This ensures fair interleaving between response batches and individual messages, preventing either from starving the other.
|
||||
The TLS handle is wrapped in an `MVar` (`newMVar h`). Both `send` (command results from `sndQ`) and `sendMsg` (messages from `msgQ`) acquire this lock via `withMVar`. This ensures fair interleaving between result batches and individual messages, preventing either from starving the other.
|
||||
|
||||
## Queue creation — ID oracle prevention
|
||||
|
||||
@@ -103,4 +103,4 @@ Every queue command calls `withQueue_` which checks if `updatedAt` matches today
|
||||
|
||||
## foldrM in client command processing
|
||||
|
||||
`foldrM process ([], [])` processes a batch of verified commands right-to-left, accumulating responses and messages. The responses list is built with `(:)`, so the final order matches the original command order. Messages from SUB are collected separately and passed as the second element of the `sndQ` tuple.
|
||||
`foldrM process ([], [])` processes a batch of verified commands right-to-left, accumulating results and messages. The results list is built with `(:)`, so the final order matches the original command order. Messages from SUB are collected separately and passed as the second element of the `sndQ` tuple.
|
||||
|
||||
@@ -24,7 +24,7 @@ The version history jumps from 12 (`blockedEntitySMPVersion`) to 14 (`proxyServe
|
||||
|
||||
`proxiedSMPRelayVersion = 18`, one below `currentClientSMPRelayVersion = 19`. The code comment states: "SMP proxy sets it to lower than its current version to prevent client version fingerprinting by the destination relays when clients upgrade at different times."
|
||||
|
||||
In practice (Server.hs), the SMP proxy uses `proxiedSMPRelayVRange` to cap the destination relay's version range in the `PKEY` response sent to the client, so the client sees a capped version range rather than the relay's actual range.
|
||||
In practice (Server.hs), the SMP proxy uses `proxiedSMPRelayVRange` to cap the destination relay's version range in the `PKEY` result sent to the client, so the client sees a capped version range rather than the relay's actual range.
|
||||
|
||||
## withTlsUnique — different API calls yield same value
|
||||
|
||||
@@ -67,7 +67,7 @@ When `clientService` is present in the client handshake, the router performs add
|
||||
- On success, the router sends `SMPServerHandshakeResponse` with a `serviceId`
|
||||
- On failure, the router sends `SMPServerHandshakeError` before raising the error
|
||||
|
||||
Per the protocol spec (v16+): "`clientService` provides long-term service client certificate for high-volume services using SMP router (chat relays, notification routers, high traffic bots). The router responds with a third handshake message containing the assigned service ID."
|
||||
Per the protocol spec (v16+): "`clientService` provides long-term service client certificate for high-volume services using SMP router (chat relays, notification routers, high traffic bots). The router returns a third handshake message containing the assigned service ID."
|
||||
|
||||
The client only includes service credentials when `v >= serviceCertsSMPVersion && certificateSent c` (the TLS client certificate was actually sent).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user