agent: use double ratchet from the first message in contact addresses, request rejection, RPC (#1831)

* agent: initialize double ratchet from the invitation via contact address (#1829) * rfc: agent support for PRC pattern without duplex connection * update rfc, plan * corrections Co-authored-by: Evgeny <evgeny@poberezkin.com> * split rfcs, add plan * update rfc * update rfc and plan * update plan * add double ratchet keys to address * agent schema * fix version in test * change ContactRequest encoding * fix tests * update plan * tests, refactor * fix compilation * refactor * refactor more * fix * add export * fix * refactor * refactor * update schema * rename * refactor * refactor 2 * comment, type * type * rename * refactor more * move type * refactor * remove comment * diff * refactor * comment * puns * get correct key * rename * add autoincrement * refactor * move * move back * simplify * rename * refactor * tuple * comment * split decryption * refactor * rename * refactor * move * remove createSndQueue * remove comment * remove JoinInvitationReq * simplify * rename * fix * type synonim * getConnData * simplify * move * encoding * version * async join with ratchet keys * fix encoding * rotate address ratchet keys * rename, parameter order * refactor, include ratchet keys into ConnectionRequestUri * simplify * remove records * refactor * rename, encoding * clean up * refactor * compatible * addrKeysE2EVersion * support IKUsePQ for contact addresses * comments, refactor condition * typos --------- Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> * agent: contact request rejection and service requests (#1833) * agent: request rejection * implement RPC requests/responses * simplify rpc, test, async * renames * clean up * fix * fix query * set flag at creation * move * command type, schema * rename * refactor * typo Co-authored-by: simplex-chat-agent[bot] <287173099+simplex-chat-agent[bot]@users.noreply.github.com> * allow overriding the service request timeout per request * fixes * use SSENT event for service replies * AgentServiceError encoding * sign service requests * add bad signature error * update the plan * clean up * refactor * empty line Co-authored-by: simplex-chat-agent[bot] <287173099+simplex-chat-agent[bot]@users.noreply.github.com> * add indices --------- Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> Co-authored-by: simplex-chat-agent[bot] <287173099+simplex-chat-agent[bot]@users.noreply.github.com> * update schemas * tryAllErrors * combine queries --------- Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> Co-authored-by: simplex-chat-agent[bot] <287173099+simplex-chat-agent[bot]@users.noreply.github.com>
2026-08-04 08:59:53 +00:00 · 2026-07-31 19:32:34 +01:00
parent 066a93861b
commit e4e5ce75fa
31 changed files with 2382 additions and 390 deletions
@@ -0,0 +1,114 @@
+---
+Proposed: 2026-07-11
+Protocol: agent-protocol (new version)
+Depends on: 2026-07-12-address-pqdr-keys
+---
+
+# One-off requests to service addresses
+
+Implementation plan: to follow, after this and the address-DR RFC are reviewed.
+
+## Problem
+
+Client applications need to interact with services, for example: badge issuance, directory requests, telemetry submissions, blockchain reads and writes, LLM calls. The only communication primitive available today is a duplex connection, so each of these interactions requires the full connection procedure - creating queues, key agreement, double ratchet initialization - and leaves persistent state on both sides: queues, ratchet state, connection records, message history.
+
+This is the wrong primitive for most service interactions:
+
+1. Cost. To send the first request to a not yet connected service, the client and the service exchange multiple commands across two servers. For "search the directory" all of it is overhead. The setup cost also creates an incentive to keep connections open, and a service with N users who used it once permanently holds N sets of queues and ratchet states.
+
+2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity.
+
+3. Encryption. Messages sent to contact addresses outside an established connection have a single layer of X25519 encryption, with no post-quantum protection and no forward secrecy. This is not acceptable for service requests.
+
+In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with `LGET`), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope.
+
+## Security objectives
+
+1. Requests from the same client must not be linkable to each other by the service or by servers, and no long term state is created on either side in the transport layer.
+2. Post-quantum resistant end-to-end encryption of requests and replies.
+3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable.
+4. A repeated request for the same operation must not be executed twice.
+
+## Solution
+
+A service address is an ordinary short-link contact address. The client sends one request to the address queue and receives replies in a reply queue it creates for the request. The double ratchet is established from the address's published keys (see the address-DR RFC): the request is the first ratchet message, and replies are subsequent ratchet messages. So a request-response exchange is a short-lived one-directional double ratchet connection, established from the first message and removed after the last.
+
+The exchange:
+
+1. Retrieve the address link data (`LGET`, via proxy when IP protection is needed): the root key, the identity key, the prekey with its id, and the KEM key.
+2. Create a reply queue (`NEW`, subscribed).
+3. Establish the sending ratchet from the published keys (`pqX3dhSnd`, `initSndRatchet`). Build the request: the reply queue, the requester's X3DH parameters, and the payload encrypted under the ratchet. Encrypt the whole request to the address queue and send it once (unauthenticated `SEND`, via proxy when IP protection is needed). There is no transport retry; a reply is the success signal, and a hard error fails the request.
+4. The service establishes the receiving ratchet from its private keys and the request's X3DH parameters (`pqX3dhRcv`, `initRcvRatchet`), decrypts the payload, and delivers it to the service application. To reply it creates a send connection with the ratchet and sends reply messages to the reply queue, each encrypted under the ratchet.
+5. The client decrypts and delivers each reply message to the application. The first reply message returns from the request; later reply messages are delivered through a callback the application registered. The exchange ends on a reply marked final. The client deletes the reply queue and the ratchet on the final message, the deadline, or when the application cancels.
+
+How this meets the objectives:
+
+1. Unlinkability: fresh X3DH keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue and both ratchets are removed after the exchange; nothing is shared between two requests.
+2. Encryption: the double ratchet with its sntrup761 KEM, from the first message.
+3. Authenticity: the ratchet is established against the identity key committed by the link hash, so a decryptable reply proves it came from the address owner; the ratchet message numbering detects dropped and reordered replies. No separate signature is needed.
+4. Single execution: the service identifies a request by the hash of its decrypted payload and, within a fixed retention period, re-sends the stored replies for a repeated request without running the operation again.
+
+## Design
+
+Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. `SMPQueueInfo`, `agentVersion`, and the X3DH and ratchet types are as in the [agent protocol](../protocol/agent-protocol.md) and `Crypto.Ratchet`.
+
+### Correlation and chat
+
+A reply is connected to its request by the reply queue: each request has its own reply queue, and every message in that queue is a reply to that request. The request hash (of the decrypted payload) is used only for idempotency. The application sets an id inside the payload to make two requests the same operation or different ones; the agent does not read it.
+
+Both ends are chat bots on the chat library over the agent. The chat library serializes a service command into the request payload and deserializes the responses; the agent transports them, establishes and removes the ratchet, and correlates by reply queue.
+
+### Request
+
+A new agent envelope, shaped like `AgentConfirmation` (X3DH parameters to establish the ratchet, plus a body encrypted under it):
+
+```abnf
+agentRequest = agentVersion %s"Q" replyQueues prekeyId sndE2EParams encRequest
+replyQueues = length 1*SMPQueueInfo ; the first is used; more are for redundancy
+prekeyId = shortString ; the published prekey used, from link data
+sndE2EParams = <requester Snd X3DH parameters, see address-DR RFC>
+encRequest = <ratchet-encrypted request payload>
+```
+
+The whole `agentRequest` is encrypted to the address queue with the per-queue layer and sent with `SEND`, as an invitation is today. The reply queue keys and the X3DH parameters are not visible to servers. A request must fit one message; a larger payload is an XFTP file description in the payload.
+
+The request hash is the SHA3-256 of the decrypted payload - the same bytes on both sides. There is no transport retry; a hard error (`AUTH`, `QUOTA`) fails the request, and the application decides whether to send a new one.
+
+### Replies
+
+The service creates a send connection with the ratchet and sends reply messages to the reply queue, each encrypted under the ratchet. A reply message uses a new agent envelope:
+
+```abnf
+agentResponse = agentVersion %s"P" final responses
+final = %s"T" / %s"F" ; T - no more reply messages follow
+responses = length 1*responseItem ; non-empty list of application responses
+responseItem = largeString ; opaque application response
+```
+
+Each message includes a list of responses, so responses known together are sent in one message and responses that become known over time are sent in separate messages. The message is encrypted and numbered by the ratchet, which authenticates it against the committed identity key and detects dropped or reordered messages; no separate signature is used.
+
+The first reply message returns from the request. Later reply messages are delivered through the callback the application registered with the request, while the process runs. The exchange ends on a message with `final = T`. The client deletes the reply queue and the ratchet on that message, on the deadline, or when the application cancels. Deleting the reply queue stops further replies. The transport keeps no exchange across a client restart; the application keeps its own state and sends a new request when it needs to.
+
+### Rejection
+
+The service refuses a request with the `AgentRejection` envelope from the [communicating rejection RFC](../../simplex-chat/docs/rfcs/2024-03-22-communicating-reject.md), sent to the reply queue under the ratchet, with an opaque application reason. The same envelope communicates refusal of a connection request, where today it is dropped silently. A rejection ends the exchange like a final reply.
+
+### Idempotency
+
+The service keeps, for a fixed retention period it chooses (1 to 24 hours, in service configuration, not in link data), the request hash, the ordered response messages it produced, and the reply queues and ratchets subscribed under that hash. A repeat request with the same hash does not reach the service application:
+
+- while the first request is being answered, the repeat establishes its own ratchet and reply queue, is added to the record, receives the responses already produced, and receives each later response too.
+- after the operation completed, the repeat receives the whole stored sequence of responses, re-encrypted under its own ratchet.
+
+The stored responses are the application response bytes, not the ratchet ciphertext, because a repeat establishes a new ratchet and the responses are re-encrypted for it. This gives single execution over at-least-once delivery. After the retention period a request with the same hash is a new operation and runs again.
+
+### Out of scope
+
+- Recovery across restart: the transport keeps no exchange across a client restart; the application persists its own state and sends a new request when it needs to.
+- Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request.
+- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion.
+- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate question, but it would fit well with name resolving to multiple addresses, both for redundancy, reliability and higher throughput.
+- Name resolution: existing addressing layer.
+
+[1]: https://tools.ietf.org/html/rfc5234
+[2]: https://tools.ietf.org/html/rfc7405
@@ -0,0 +1,83 @@
+---
+Proposed: 2026-07-12
+Protocol: agent-protocol (new version)
+---
+
+# Establishing the double ratchet from address data
+
+## Problem
+
+A contact address receives a connection request as an `AgentInvitation` message, encrypted only with the per-queue X25519 layer. The double ratchet is established later: the address owner joins the requester's connection request, generates its X3DH keys, and sends them in the confirmation. So the first message to an address - the invitation and the profile in it - is not under the double ratchet, and has no post-quantum protection.
+
+The cause is that the address owner's X3DH contribution is generated per request and sent in the confirmation, so it cannot exist before the requester's first message.
+
+## Solution
+
+Publish the address owner's X3DH contribution in the address link data, so a requester can establish the double ratchet in its first message. The requester runs the existing `pqX3dhSnd` against the published keys, initializes a sending ratchet, and encrypts its first message under it. The owner runs the existing `pqX3dhRcv` against its stored private keys and the requester's X3DH keys from the message, initializes a receiving ratchet, and decrypts it.
+
+Publish the owner's X3DH contribution - two X448 keys and an optional sntrup761 KEM key, with the e2e version range - as one bundle in the mutable contact user data, signed by the root key. The bundle is the existing `RcvE2ERatchetParamsUri` type that a one-time invitation already advertises, with an id for rotation.
+
+This is backward compatible. A requester that does not use the published bundle sends a current `AgentInvitation` with its own X3DH keys, and the owner does what it does today: generates fresh X3DH keys and sends them in the confirmation. The owner branches on whether the incoming message uses the published bundle.
+
+Three properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. A decryptable message proves the sender established X3DH against the root-signed keys, so it authenticates the address owner without a separate signature. And because the bundle is in mutable data, an existing address can advertise the double ratchet by updating its mutable data - no new link.
+
+## Design
+
+Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. Key and ratchet types are as in `Crypto.Ratchet` and the [agent protocol](../protocol/agent-protocol.md); all DH keys are X448, the KEM is sntrup761.
+
+### Published keys in link data
+
+The owner's X3DH contribution is a ratchet-keys bundle appended to the mutable contact user data, signed by the root key. Nothing is added to the immutable fixed link data:
+
+```abnf
+userContactData =/ ratchetKeys ; appended, ignored by earlier versions
+ratchetKeys = %s"0" / (%s"1" ratchetKeyId e2eParams)
+ratchetKeyId = shortString ; identifies this bundle, changes on rotation, echoed in the request
+e2eParams = <RcvE2ERatchetParamsUri: e2e version range, two X448 keys, optional sntrup761 key>
+```
+
+`e2eParams` is the existing `RcvE2ERatchetParamsUri` - the same type a one-time invitation advertises - so a requester reads it, negotiates the concrete e2e version against its own range, and runs `pqX3dhSnd` against it, exactly as it does for an invitation. The KEM key is optional (the params' KEM field is a `Maybe`), controlled by the address's initial-keys mode, the same 3-way choice as invitations: advertise the KEM (post-quantum from the first message), advertise X448-only but still support post-quantum if the requester proposes its own KEM (one message later, avoiding the ~1158-byte key in link data), or no post-quantum.
+
+The owner keeps the private side of each bundle - the two X448 private keys and, when PQ is on, the KEM keypair - indexed by `ratchetKeyId`. On rotation with `LSET` it publishes a new bundle with a new `ratchetKeyId` and keeps the previous private keys for a window covering queue message retention, so a request that used a just-rotated bundle still decrypts. Because the bundle is in mutable data, an existing address advertises the double ratchet by updating its mutable data - no new link, no re-creation.
+
+### Request confirmation
+
+A requester that uses the published keys establishes the sending ratchet before its first message, so it sends that message as a confirmation, not an invitation - the same envelope a joining party sends in a connection. The confirmation gains an optional `ratchetKeyId` naming the bundle the requester used, so the owner selects the matching private keys:
+
+```abnf
+agentConfirmation =/ ratchetKeyId ; the bundle the requester used, echoed; absent on other confirmations
+```
+
+The confirmation holds the requester's Snd X3DH parameters (so the owner runs `pqX3dhRcv`) and, encrypted under the ratchet, the first message. A confirmation with `ratchetKeyId` on a contact address takes the published-key path; an `AgentInvitation`, as today, takes the current path where the owner generates fresh X3DH keys and returns them in its own confirmation. A connection-request URI is unchanged: it advertises the requester's Rcv parameters and must not include Snd parameters.
+
+### Establishing the ratchet
+
+Requester:
+
+1. Retrieve link data (`LGET`), read the bundle - its `ratchetKeyId` and `e2eParams` (`RcvE2ERatchetParamsUri`) - and negotiate the concrete e2e version against its own range.
+2. `generateSndE2EParams` for its own X3DH contribution - encapsulating to the bundle's KEM if it advertises one, or proposing its own KEM if the requester wants post-quantum and the bundle is X448-only.
+3. `pqX3dhSnd` against the bundle's parameters, then `initSndRatchet` - the sending ratchet.
+4. Encrypt the first message under the ratchet, and send a confirmation with its Snd parameters and `ratchetKeyId`.
+
+Owner:
+
+1. On a confirmation with `ratchetKeyId` on a contact address, select the private X3DH keys and, if any, KEM keypair by `ratchetKeyId` (current or a retained previous generation).
+2. `pqX3dhRcv` against the requester's Snd parameters with those private keys, then `initRcvRatchet` - the receiving ratchet. Decrypting the first message advances the ratchet and gives it a send side, so the owner can reply.
+3. Decrypt, and reply under the ratchet.
+
+A request whose `ratchetKeyId` is no longer retained cannot be decrypted; the owner does not learn the requester or its reply address, and the requester's attempt fails at its own timeout.
+
+### Authentication
+
+The ratchet-keys bundle is in the mutable link data, signed by the root key. A decryptable message proves the sender established X3DH against those root-signed keys: an SMP server cannot substitute them without forging the root signature (`decryptLinkData` verifies it), which is the X3DH anti-substitution property. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The root Ed25519 key is the address's signing identity; the X3DH keys are separate DH keys (X448), and X3DH is over crypto_box, so deniability is preserved. A malicious server can still serve an older but validly-signed bundle (rollback to a retired generation); this is bounded by the retention window and by the ratchet advancing after the first message, and a per-key signature would not prevent it. Reusing a bundle across requesters is consistent with the address already being a shared identifier.
+
+## Uses
+
+- Invitations to an address, and the profile in them, are under the double ratchet from the first message.
+- An existing address gains the double ratchet when the app updates its mutable link data (`LSET`, with the user's confirmation and current profile, combined with the full→short address migration); no new link is issued.
+- The service RPC (see the RPC RFC) establishes the ratchet this way to send the request as the first ratchet message.
+
+This RFC depends on nothing else here. It replaces the need for the PQ-queue RFC in the address case, because the ratchet provides post-quantum protection for the first message; the PQ-queue RFC remains for first messages that do not establish a ratchet.
+
+[1]: https://tools.ietf.org/html/rfc5234
+[2]: https://tools.ietf.org/html/rfc7405
@@ -0,0 +1,42 @@
+---
+Proposed: 2026-07-12
+Protocol: smp-client (new version)
+---
+
+# Post-quantum encryption of the SMP queue layer
+
+## Problem
+
+A message sent to a queue outside an established double ratchet has one layer of end-to-end encryption: NaCl crypto_box over an X25519 DH secret. The sender generates an ephemeral X25519 key, computes the secret with the recipient's per-queue DH key, and puts its ephemeral key in the message public header (`agentCbEncryptOnce`). This layer protects invitations, confirmations, and the profile sent with them.
+
+It is not post-quantum. An adversary that records this traffic and later has a quantum computer can recover the X25519 secret and decrypt it. The double ratchet adds a post-quantum KEM once it is established, but the first message to a queue, before the ratchet, has only X25519.
+
+This RFC adds post-quantum protection to the single-shot queue encryption itself, for cases that do not establish a double ratchet from the first message. Where a double ratchet is established from the first message (see the address-DR RFC), the ratchet provides post-quantum protection and this layer is not needed.
+
+## Solution
+
+Extend the single-shot queue encryption to a hybrid X25519 + sntrup761 scheme, in a new SMP client version. The recipient publishes a KEM encapsulation key alongside its per-queue DH key. The sender encapsulates to it, combines the KEM shared secret with the X25519 DH secret, and encrypts the body with the combined secret. The KEM ciphertext travels in the message public header next to the ephemeral X25519 key.
+
+Recording the traffic and breaking X25519 later is not sufficient: without breaking sntrup761 as well, the combined secret is not recoverable.
+
+## Design
+
+The message public header (`PubHeader`) gains a hybrid variant, selected by a version and a tag, so older senders and the empty-header case are unchanged:
+
+```abnf
+smpPubHeaderHybrid = smpClientVersion %s"2" senderPublicDhKey kemCiphertext
+senderPublicDhKey = length x509encoded ; sender ephemeral X25519 key
+kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes
+```
+
+The secret combines both shared secrets, and the body is encrypted with NaCl secret_box (the DH-only path uses crypto_box today; the combined secret is no longer a plain DH result, so it is used as a secret_box key), padded to the same lengths:
+
+```
+secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret)
+```
+
+The recipient's KEM encapsulation key is distributed the same way its per-queue DH key is today - in the queue address for a connection request, and in link data for a short link. The KEM ciphertext is stored the same way the ephemeral DH key is: in the public header, readable by the destination server (and not by the proxy with proxied sending), which cannot derive the secret without the recipient's KEM private key.
+
+The recipient stores the computed secret the way the per-queue DH secret is stored on receiving the first message (`setRcvQueueConfirmedE2E`), and reuses it for later messages on the queue.
+
+Sizes: the KEM ciphertext is 1039 bytes and the encapsulation key 1158 bytes. Link data user data is padded to 13784 bytes, so the key fits with application data. This RFC is independent of the RPC, SSND, and address-DR RFCs.
@@ -0,0 +1,55 @@
+---
+Proposed: 2026-07-12
+Protocol: smp (new version)
+---
+
+# SSND: combined secure-and-send command
+
+## Problem
+
+Two SMP flows secure a messaging queue and then immediately send the first message to it, as two commands and two round trips:
+
+- The fast connection handshake: the joining party secures the queue with `SKEY`, then sends the confirmation with `SEND`.
+- Any first send to a sender-securable queue where the sender both secures it and delivers the first message.
+
+The two commands express one intent - "this is my key, and here is my first message" - so they can be one command and one round trip. The combination must be idempotent, because the first send is retried on network failure and a queue is often secured before the response is known. `SKEY` is already idempotent (a repeat with the same key succeeds). `SEND` is not, so a naive combination would deliver a duplicate message on retry.
+
+## Solution
+
+A new command `SSND` combines `SKEY` and `SEND` in one transmission, idempotent in both parts:
+
+- Key part: as `SKEY` - a repeat with the same key succeeds, a different key fails with `AUTH`.
+- Send part: the server keeps the hash of the message until it is acknowledged, and reports a repeat of the same message as delivered without delivering it again.
+
+The server-side hash covers the common case, when the retry arrives before the message is acknowledged. A retry that arrives after the acknowledgement is delivered as a duplicate and discarded by the receiving agent by message hash, as duplicate messages are discarded today.
+
+## Design
+
+Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. `senderAuthPublicKey`, `msgFlags` and `smpEncMessage` are as in the [SMP protocol](../protocol/simplex-messaging.md).
+
+```abnf
+secureSend = %s"SSND " senderAuthPublicKey SP msgFlags SP smpEncMessage
+senderAuthPublicKey = length x509encoded
+```
+
+`SSND` is a sender command, authorized with the key it sets, and accepted only on messaging-mode queues (`QMMessaging`), where the sender can secure the queue. The server responds `OK` or `ERR`.
+
+Server processing:
+
+1. Secure the queue with the key, as `SKEY`. A repeat with the same key succeeds; a different key returns `AUTH`.
+2. If the queue holds an unacknowledged message whose stored hash equals the hash of this message, respond `OK` without storing it again.
+3. Otherwise store the message, keep its hash with the queue until the message is acknowledged, and deliver it.
+
+The stored hash is one value per not-yet-acknowledged queue message. It is removed when the message is acknowledged. All queue store backends (in-memory, journal, PostgreSQL) keep it.
+
+`SSND` composes with the proxy protocol without change: `proxySMPCommand` forwards any sender command through `PFWD`/`RFWD`, so `SSND` is proxied as `SEND` and `SKEY` are today.
+
+## Uses
+
+- The fast connection handshake replaces `SKEY` then the `SEND` confirmation with one `SSND`.
+- The service RPC response (see the RPC RFC) secures the reply queue and sends the first reply with one `SSND`.
+
+This RFC is independent of the RPC, PQ-queue, and address-DR RFCs and can be implemented on its own.
+
+[1]: https://tools.ietf.org/html/rfc5234
+[2]: https://tools.ietf.org/html/rfc7405