From 3c5752383d450b4f310b16ed84a90ad3a2fa1bcb Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Mon, 9 Mar 2026 16:29:05 +0000 Subject: [PATCH] update protocol docs --- protocol/agent-protocol.md | 46 ++++++++++++++++---- protocol/push-notifications.md | 28 +++++++++++-- protocol/simplex-messaging.md | 77 ++++++++++++++++++++-------------- protocol/xrcp.md | 4 +- 4 files changed, 110 insertions(+), 45 deletions(-) diff --git a/protocol/agent-protocol.md b/protocol/agent-protocol.md index f43d195e4..83898d9b9 100644 --- a/protocol/agent-protocol.md +++ b/protocol/agent-protocol.md @@ -177,7 +177,7 @@ These messages are encrypted with per-queue shared secret using NaCL crypto_box decryptedSMPClientMessage = agentConfirmation / agentMsgEnvelope / agentInvitation / agentRatchetKey agentConfirmation = agentVersion %s"C" ("0" / "1" sndE2EEncryptionParams) encConnInfo agentVersion = 2*2 OCTET -sndE2EEncryptionParams = TODO +sndE2EEncryptionParams = encConnInfo = doubleRatchetEncryptedMessage agentMsgEnvelope = agentVersion %s"M" encAgentMessage @@ -187,11 +187,20 @@ agentInvitation = agentVersion %s"I" connReqLength connReq connInfo connReqLength = 2*2 OCTET ; Word16 agentRatchetKey = agentVersion %s"R" rcvE2EEncryptionParams agentRatchetInfo -rcvE2EEncryptionParams = TODO +rcvE2EEncryptionParams = -doubleRatchetEncryptedMessage = TODO +doubleRatchetEncryptedMessage = ``` +The maximum size of the encrypted connection info and agent message depend on whether post-quantum key exchange is used: + +| Constant | PQ on | PQ off | +|----------|-------|--------| +| `e2eEncConnInfoLength` | 11106 | 14832 | +| `e2eEncAgentMsgLength` | 13618 | 15840 | + +The PQ-on sizes are smaller because the ratchet header and reply link include larger PQ keys (SNTRUP761). + This syntax of decrypted SMP client message body is defined by `decryptedAgentMessage` below. Decrypted SMP message client body can be one of 4 types: @@ -465,10 +474,20 @@ fixedData = agentVersionRange rootKey linkConnReq [linkEntityId] agentVersionRange = version version ; min and max agent protocol version version = 2*2 OCTET rootKey = length x509encoded ; Ed25519 public key +linkConnReq = connectionRequestUri ; see full connection link syntax above linkEntityId = shortString userData = invitationLinkData / contactLinkData -invitationLinkData = %s"I" agentVersionRange connInfo +invitationLinkData = %s"I" agentVersionRange userLinkData contactLinkData = %s"C" agentVersionRange userContactData +userLinkData = *OCTET ; opaque application data (e.g., user profile) +userContactData = direct ownersList relaysList userLinkData +direct = %s"T" / %s"F" ; whether direct connection via connReq is allowed +ownersList = length *ownerAuth +ownerAuth = shortString ; length-prefixed encoding of (ownerId ownerKey authOwnerSig) +ownerId = shortString ; application-specific owner ID (e.g., MemberId) +ownerKey = length x509encoded ; Ed25519 public key +authOwnerSig = 64*64 OCTET ; Ed25519 signature of (ownerId || ownerKey) by previous owner +relaysList = length *connShortLink ; alternative relay short links largeString = 2*2 OCTET *OCTET ; Word16 length prefix length = 1*1 OCTET shortString = length *OCTET @@ -564,6 +583,14 @@ This api is also used to acknowledge message delivery to the sending party - tha `getNotificationMessage` is used by push notification subsystem of the client application to receive the message from a specific messaging queue mentioned in the notification. The client application would receive `MSG` and any other events from the agent, and then `MSGNTF` event once the message related to this notification is received. +#### Set short link data + +`setConnectionLink` api (`LSET` command) is used to set or update short link data associated with a contact address queue. Returns `LINK` event with the short link URI. + +#### Get short link data + +`getConnectionLink` api (`LGET` command) is used to retrieve and decrypt the short link data from the server. Returns `LDATA` event with the decrypted link data. + #### Rotate message queue to another server `switchConnection` api is used to rotate connection queues to another messaging server. @@ -574,7 +601,7 @@ This api is also used to acknowledge message delivery to the sending party - tha #### Delete connection -`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQ` and `DEL_CONN` events. +`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQS` and `DEL_CONNS` events. #### Suspend connection @@ -601,8 +628,13 @@ Agent API uses these events dispatch to notify client application about events r - `MSGNTF` - sent after agent received and processed the message referenced in the push notification. - `RCVD` - notification confirming message receipt by another party. - `QCONT` - notification that the agent continued sending messages after queue capacity was exceeded and recipient received all messages. -- `DEL_RCVQ` - confirmation that message queue was deleted. -- `DEL_CONN` - confirmation that connection was deleted. +- `LINK` - short link URI created or updated for a contact address. +- `LDATA` - decrypted short link data received from the server. +- `DELD` - notification that the connection was deleted. +- `JOINED` - notification that a member joined via a contact address. +- `STAT` - connection statistics event. +- `DEL_RCVQS` - confirmation that receiver message queues were deleted. +- `DEL_CONNS` - confirmation that connections were deleted. - `OK` - confirmation that asynchronous api call was successful. - `ERR` - error of asynchronous api call or some other error event. diff --git a/protocol/push-notifications.md b/protocol/push-notifications.md index 6d5e1dea0..c5b2ce154 100644 --- a/protocol/push-notifications.md +++ b/protocol/push-notifications.md @@ -1,7 +1,12 @@ -Version 2, 2024-06-22 +Version 3, 2025-01-24 # Overview of push notifications for SimpleX Messaging Servers +This document describes Notification Server protocol version 3. Version history: +- v1: initial version +- v2: authenticated commands, command batching +- v3: detailed invalid token reason + ## Table of contents - [Introduction](#introduction) @@ -91,13 +96,15 @@ To manage notification subscriptions to SMP servers, SimpleX Notification Server This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the server certificate is not included in the handshake). +The client and server use ALPN extension with `ntf/1` protocol name to agree handshake version. + Protocol commands have this syntax: ``` ntfServerTransmission = ntfServerCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd / replaceTokenCmd / deleteTokenCmd / cronCmd / - newSubCmd / checkSubCmd / deleteSubCmd + newSubCmd / checkSubCmd / deleteSubCmd / pingCmd ``` ### Register new notification token @@ -159,7 +166,9 @@ The response to this command: ```abnf tokenStatusResp = %s"TKN" SP tokenStatus -tokenStatus = %s"NEW" / %s"REGISTERED" / %s"INVALID" / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED" +tokenStatus = %s"NEW" / %s"REGISTERED" / tokenInvalid / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED" +tokenInvalid = %s"INVALID" ["," invalidReason] ; optional reason added in v3 +invalidReason = %s"BAD" / %s"TOPIC" / %s"EXPIRED" / %s"UNREGISTERED" ``` ### Replace notification token @@ -249,7 +258,7 @@ The response: subStatusResp = %s"SUB" SP subStatus subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP server disconnect/timeout while ntf server is retrying to connect %s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another server subscribed to notifications - %s"AUTH" / subErrStatus + %s"AUTH" / %s"DELETED" / %s"SERVICE" / subErrStatus subErrStatus = %s"ERR" SP shortString ``` @@ -265,6 +274,17 @@ The response to this command is `okResp` or `errorResp`. After this command no more message notifications will be sent from this queue. +### Keep-alive command + +To keep the transport connection alive the clients should use `PING` command: + +```abnf +pingCmd = %s"PING" +pongResp = %s"PONG" +``` + +This command is sent unsigned and without entity ID. + ### Error responses All commands can return error response: diff --git a/protocol/simplex-messaging.md b/protocol/simplex-messaging.md index 5a077eff2..758d4cead 100644 --- a/protocol/simplex-messaging.md +++ b/protocol/simplex-messaging.md @@ -102,7 +102,7 @@ This document describes SMP protocol version 19. Versions 1-5 are discontinued. - v16: service certificates - v17: create notification credentials with NEW command - v18: support client notices in BLOCKED error -- v19: service subscriptions to messages (SUBS, SOKS, ENDS commands) +- v19: service subscriptions to messages (SUBS, NSUBS, SOKS, ENDS, ALLS commands) ## Introduction @@ -445,11 +445,13 @@ SMP protocol supports client services - high capacity clients that act as servic ### Service roles -A client service can have one of two roles: +A client service can have one of three roles: -- **Messaging** - Message receiver service that subscribes to and receives messages from multiple SMP queues with a single command. +- **Messaging** (`"M"`) - Message receiver service that subscribes to and receives messages from multiple SMP queues with a single command. -- **Notifications** - Notification service that subscribes to queue notifications and delivers push notifications to user devices. +- **Notifications** (`"N"`) - Notification service that subscribes to queue notifications and delivers push notifications to user devices. + +- **Proxy** (`"P"`) - Proxy service that forwards sender commands to destination servers. Service role is identified in the transport handshake and determines what commands the service is authorized to send. @@ -465,7 +467,7 @@ Service certificates are included in the client handshake and verified by the se ```abnf clientHandshakeService = serviceRole serviceCertKey -serviceRole = %s"M" / %s"N" ; Messaging / Notifier +serviceRole = %s"M" / %s"N" / %s"P" ; Messaging / Notifier / Proxy serviceCertKey = certChainPubKey ``` @@ -506,7 +508,7 @@ transmissionCount = 1*1 OCTET ; equal or greater than 1 transmissions = transmissionLength transmission [transmissions] transmissionLength = 2*2 OCTET ; word16 encoded in network byte order -transmission = authorization authorized +transmission = authorization [serviceSig] authorized authorized = sessionIdentifier corrId entityId smpCommand corrId = %x18 24*24 OCTET / %x0 "" ; corrId is required in client commands and server responses, @@ -517,6 +519,8 @@ entityId = shortString ; queueId or proxySessionId authorization = shortString ; signature or authenticator ; empty authorization can be used with "send" before the queue is secured with secure command ; authorization is always empty with "ping" and server responses +serviceSig = shortString ; optional Ed25519 service signature (v16+) + ; present only in service sessions when authorization is non-empty sessionIdentifier = "" ; sessionIdentifierForAuth = shortString ; sessionIdentifierForAuth MUST be included in authorized transmission body. @@ -881,17 +885,17 @@ This command is sent to the server by the sender both to confirm the queue after send = %s"SEND " msgFlags SP smpEncMessage msgFlags = notificationFlag reserved notificationFlag = %s"T" / %s"F" -smpEncMessage = smpEncClientMessage / smpEncConfirmation ; message up to 16064 bytes +smpEncMessage = smpEncClientMessage / smpEncConfirmation ; message up to 16048 bytes (v11+) -smpEncClientMessage = smpPubHeaderNoKey msgNonce sentClientMsgBody ; message up to 16064 bytes +smpEncClientMessage = smpPubHeaderNoKey msgNonce sentClientMsgBody ; message up to maxMessageLength bytes smpPubHeaderNoKey = smpClientVersion "0" -sentClientMsgBody = 16016*16016 OCTET +sentClientMsgBody = 16000*16000 OCTET ; = maxMessageLength(v11+) - 48 = 16048 - 48 smpEncConfirmation = smpPubHeaderWithKey msgNonce sentConfirmationBody smpPubHeaderWithKey = smpClientVersion "1" senderPublicDhKey ; sender's Curve25519 public key to agree DH secret for E2E encryption in this queue ; it is only sent in confirmation message -sentConfirmationBody = 15920*15920 OCTET ; E2E-encrypted smpClientMessage padded to 16016 bytes before encryption +sentConfirmationBody = 15904*15904 OCTET ; E2E-encrypted smpClientMessage padded to e2eEncMessageLength before encryption senderPublicDhKey = length x509encoded smpClientVersion = word16 @@ -917,16 +921,18 @@ Until the queue is secured, the server should accept any number of unsigned mess The body should be encrypted with the shared secret based on recipient's "public" key (`EK`); once decrypted it must have this format: ```abnf -sentClientMsgBody = +sentClientMsgBody = + ; e2eEncMessageLength = 16000 smpClientMessage = emptyHeader clientMsgBody emptyHeader = "_" -clientMsgBody = *OCTET ; up to 16016 - 2 +clientMsgBody = *OCTET ; up to e2eEncMessageLength - 2 -sentConfirmationBody = +sentConfirmationBody = + ; e2eEncConfirmationLength = 15904 smpConfirmation = smpConfirmationHeader confirmationBody smpConfirmationHeader = emptyHeader / %s"K" senderKey ; emptyHeader is used when queue is already secured by sender -confirmationBody = *OCTET ; up to 15920 - 2 +confirmationBody = *OCTET ; up to e2eEncConfirmationLength - 2 senderKey = length x509encoded ; the sender's Ed25519 or X25519 public key to authorize SEND commands for this queue ``` @@ -940,15 +946,15 @@ SMP transmission structure for directly sent messages: 1 | transmission count (= 1) 2 | originalLength 299- | authorization sessionId corrId queueId %s"SEND" SP (1+114 + 1+32? + 1+24 + 1+24 + 4+1 = 203) - ....... smpEncMessage (= 16064 bytes = 16384 - 320 bytes) + ....... smpEncMessage (= 16048 bytes for v11+, within 16384 - 320 bytes) 8- | smpPubHeader (for messages it is only version and '0' to mean "no DH key" = 3 bytes) 24 | nonce for smpClientMessage 16 | auth tag for smpClientMessage - ------- smpClientMessage (E2E encrypted, = 16016 bytes = 16064 - 48) + ------- smpClientMessage (E2E encrypted, = 16000 bytes = 16048 - 48, for v11+) 2 | originalLength 2- | smpPrivHeader ....... - | clientMsgBody (<= 16012 bytes = 16016 - 4) + | clientMsgBody (<= 15996 bytes = 16000 - 4) ....... 0+ | smpClientMessage pad ------- smpClientMessage end @@ -971,17 +977,17 @@ SMP transmission structure for received messages: 2 | originalLength 8 | timestamp 8- | message flags - ....... smpEncMessage (= 16064 bytes = 16082 - 18 bytes) + ....... smpEncMessage (= 16048 bytes for v11+, padded within 16082 - 18 = 16064 bytes) 8- | smpPubHeader (empty header for the message) 24 | nonce for smpClientMessage 16 | auth tag for smpClientMessage - ------- smpClientMessage (E2E encrypted, = 16016 bytes = 16064 - 48 bytes) + ------- smpClientMessage (E2E encrypted, = 16000 bytes = 16048 - 48 bytes, for v11+) 2 | originalLength 2- | smpPrivHeader (empty header for the message) - ....... clientMsgBody (<= 16012 bytes = 16016 - 4) + ....... clientMsgBody (<= 15996 bytes = 16000 - 4) -- TODO move internal structure (below) to agent protocol 20- | agentPublicHeader (the size is for user messages post handshake, without E2E X3DH keys - it is version and 'M' for the messages - 3 bytes in total) - ....... E2E double-ratchet encrypted (<= 15996 bytes = 16016 - 20) + ....... E2E double-ratchet encrypted (<= 15980 bytes = 16000 - 20) 1 | encoded double ratchet header length (it is 123 now) 123 | encoded double ratchet header, including: 2 | version @@ -989,12 +995,12 @@ SMP transmission structure for received messages: 16 | double-ratchet header auth tag 1+88 | double-ratchet header (actual size is 69 bytes, the rest is reserved) 16 | message auth tag (IV generated from chain ratchet) - ------- encrypted agent message (= 15856 bytes = 15996 - 140) + ------- encrypted agent message (= 15840 bytes = 15980 - 140) 2 | originalLength 64- | agentHeader (the actual size is 41 = 8 + 1+32) 2 | %s"MM" ....... - | application message (<= 15788 bytes = 15856 - 68) + | application message (<= 15772 bytes = 15840 - 68) ....... 0+ | encrypted agent message pad ------- encrypted agent message end @@ -1117,7 +1123,7 @@ Transmission sent to proxy server should use session ID as entity ID and use a r Encrypted transmission should use the received session ID from the connection between proxy server and destination server in the authorized body. ```abnf -proxyCommand = %s"PFWD" SP smpVersion commandKey +proxyCommand = %s"PFWD" SP smpVersion commandKey smpVersion = 2*2 OCTET commandKey = length x509encoded ``` @@ -1127,7 +1133,7 @@ The proxy server will forward the encrypted transmission in `RFWD` command (see Having received the `RRES` response from the destination server, proxy server will forward `PRES` response to the client. `PRES` response should use the same correlation ID as `PFWD` command. The destination server will use this correlation ID increased by 1 as a nonce for encryption of the response. ```abnf -proxyResponse = %s"PRES" SP +proxyResponse = %s"PRES" SP ``` #### Forward command to destination server @@ -1158,7 +1164,7 @@ relayResponse = %s"RRES" SP ### Short link commands -These commands are used by senders to access queues via short links (added in v8). +These commands are used by senders to access queues via short links (added in v15). #### Set link key @@ -1270,7 +1276,7 @@ The server must deliver messages to all subscribed simplex queues on the current message = %s"MSG" SP msgId encryptedRcvMsgBody encryptedRcvMsgBody = ; server-encrypted padded sent msgBody - ; maxMessageLength = 16064 + ; maxMessageLength = 16048 (v11+) rcvMsgBody = timestamp msgFlags SP sentMsgBody / msgQuotaExceeded msgQuotaExceeded = %s"QUOTA" SP timestamp msgId = length 24*24OCTET @@ -1367,16 +1373,16 @@ deleted = %s"DELD" - `IDENTITY` - incorrect server identity (certificate fingerprint does not match server address). - `BAD_AUTH` - incorrect or missing server credentials in handshake. - authentication error (`AUTH`) - incorrect authorization, unknown (or suspended) queue, sender's ID is used in place of recipient's and vice versa, and some other cases (see [Send message](#send-message) command). -- blocked entity error (`BLOCKED`) - the entity (queue or message) was blocked due to policy violation (added in v17). Contains blocking information: +- blocked entity error (`BLOCKED`) - the entity (queue or message) was blocked due to policy violation (added in v12). Contains blocking information: - `reason` - blocking reason (`spam` or `content`). - `notice` - optional client notice with additional information. - service error (`SERVICE`) - service-related error. - crypto error (`CRYPTO`) - cryptographic operation failed. - message queue quota exceeded error (`QUOTA`) - too many messages were sent to the message queue. Further messages can only be sent after the recipient retrieves the messages. - store error (`STORE`) - server storage error with error message. -- message expired (`EXPIRED`) - message has expired. +- relay public key expired (`EXPIRED`) - relay public key has expired. - no message (`NO_MSG`) - no message available or message ID mismatch. -- sent message is too large (> 16064) to be delivered (`LARGE_MSG`). +- sent message is too large (> maxMessageLength) to be delivered (`LARGE_MSG`). - internal server error (`INTERNAL`). - duplicate error (`DUPLICATE_`) - internal duplicate detection error (not returned by server). @@ -1513,7 +1519,7 @@ clientKey = length x509encoded ; X25519 public key for session encryption - only proxyServer = %s"T" / %s"F" ; true if connecting client is a proxy server optClientService = %s"0" / (%s"1" clientService) ; optional service client credentials clientService = serviceRole serviceCertKey -serviceRole = %s"M" / %s"N" ; Messaging / Notifier +serviceRole = %s"M" / %s"N" / %s"P" ; Messaging / Notifier / Proxy serviceCertKey = certChain signedServiceKey signedServiceKey = originalLength x509encoded ; Ed25519 key signed by service certificate @@ -1530,7 +1536,14 @@ pad = *OCTET `proxyServer` flag (v14+) disables additional transport encryption inside TLS for proxy connections, since proxy server connection already has additional encryption. -`clientService` (v16+) provides long-term service client certificate for high-volume services using SMP server (chat relays, notification servers, high traffic bots). The server responds with a third handshake message containing the assigned service ID. +`clientService` (v16+) provides long-term service client certificate for high-volume services using SMP server (chat relays, notification servers, high traffic bots). The server responds with a third handshake message containing the assigned service ID: + +```abnf +paddedServerHandshakeResponse = +serverHandshakeResponse = %s"R" serviceId / %s"E" handshakeError +serviceId = shortString +handshakeError = transportError +``` `ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and servers must ignore any extra bytes within the original block length. diff --git a/protocol/xrcp.md b/protocol/xrcp.md index c8042f858..b3a304572 100644 --- a/protocol/xrcp.md +++ b/protocol/xrcp.md @@ -143,7 +143,7 @@ hostHello = %s"HELLO " dhPubKey nonce encrypted(unpaddedSize hostHelloJSON hello unpaddedSize = largeLength dhPubKey = length x509encoded pad = -helloPad = +helloPad = largeLength = 2*2 OCTET ``` @@ -190,7 +190,7 @@ ctrlHello = %s"HELLO " kemCiphertext encrypted(unpaddedSize ctrlHelloJSON helloP unpaddedSize = largeLength kemCiphertext = largeLength *OCTET pad = -helloPad = +helloPad = largeLength = 2*2 OCTET ctrlError = %s"ERROR " nonce encrypted(unpaddedSize ctrlErrorMessage helloPad) pad