dandri/simplexmq
1
0
Fork 0
mirror of https://github.com/simplex-chat/simplexmq.git synced 2026-04-23 10:16:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

rfc: protocol for SMP queue data blobs and short links (#1309)

Browse Source 
* rfc: protocol for SMP queue data blobs and short links

* update

* private rendezvous
This commit is contained in:
Evgeny
2024-09-24 12:29:11 +01:00
committed by GitHub
parent 7dcac19a67
commit d0ee505c3a
4 changed files with 311 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
rfcs/2024-06-21-short-links.md Normal file
View File

@@ -0,0 +1,124 @@
# Short invitation links
## Problem
Long links look scary and unsafe for many users. While this is a perceived problem, rather than a real one, it hurts adoption.
What is worse, long links do not fit in profile descriptions of other social networks where people might want to advertize their contact addresses.
The current link size limitation is also the reason for not including PQ KEM keys into invitation links and addresses, postponing the moment when PQ-resistant encryption kicks in - if we include PQ KEM key into the link, the QR code will not be scannable.
Additionally, if we store short links, they can also include chat preferences and public profile data.
## Solution
MITM-resistant link shortening.
Instead of generating the random address that would resolve into the link - doing so would create the possibility of MITM by the server hosting this link - we can use private key as the link ID that will be passed to the accepting party, and the hash of the public key as ID for the server - the accepting party would present this key itself as ID and it will also be used for server to client encryption (see Protocol below). HKDF will be used to derive symmetric key from private key and used in secret_box together with random nonce (to allow replacing data with the same key but with a different nonce - nonce will be sent to the server too). secret_box construction is authenticated encryption, so it would protect from MITM.
The proposed syntax:
```abnf
shortConnectionRequest = connectionScheme "/" connReqType "#/" smpServer "/" linkHash
connReqType = %s"invitation" / %s"contact"
connectionScheme = (%s"https://" clientAppServer) / %s"simplex:"
clientAppServer = hostname [ ":" port ]
; client app server, e.g. simplex.chat
smpServer = serverIdentity "@" srvHosts [":" port] ; no smp:// prefix, no escaping
srvHosts = <hostname> ["," srvHosts] ; RFC1123, RFC5891
linkHash = <base64url encoded SHA256 or SHA512 hash of the original link>
```
If SMP server supports pages, its name can be used as clientAppServer, without repeating it after #, for a shorter link.
Example link:
```
https://simplex.chat/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU=@smp8.simplex.im/abcdefghij0123456789abcdefghij0123456789abc=
```
This link has the length of ~136 characters (256 bits), which is shorter than the full contact address (~310 characters) and much shorter than invitation links (~528 characters) even without post-quantum keys added to them.
This size can be further reduced by
- use server domain in the link.
- do not include onion address, as the connection happens via proxy anyway, if it's untrusted server.
- not pinning server TLS certificate - the downside here is that while the attack that compromises TLS will not be able to substitute the link (because it's hash will not match), it will be able to intercept and to block it.
- using shorter hash, e.g. SHA128 - reducing the collision resistance.
If the server is known, the client could use it's hash and onion address, otherwise it could trust the proxy to use any existing session with the same hostname or to accept the risk of interception - given that there is no risk of substitution.
With the first two of these "improvements" the link could be ~122 characters:
```
https://smp8.simplex.im/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU@/abcdefghij0123456789abcdefghij0123456789abc
```
If onion address is preserved the link will be ~184 characters (won't fit in Twitter 160 characters bio):
```
https://smp8.simplex.im/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU@beccx4yfxxbvyhqypaavemqurytl6hozr47wfc7uuecacjqdvwpw2xid.onion/abcdefghij0123456789abcdefghij0123456789abc
```
If we implement it, the request to resolve the link would be made via proxied SMP command (to avoid the direct connection between the client and the recipient's server).
Pros:
- a bit shorter link.
- possibility to include post-quantum keys into the full link keeping the same shortened link size.
- possibility to include chat profile of contact or group, and preferences, for a much better connection experience, and to show this information when the link sent in the conversation (clients can resolve them automatically, without connecting - it can be resolved by the sending clients).
- server will not have access to the link.
Cons:
- protocol complexity.
- observers can access the link content, so for 1-time invitation we should only include permissions and not profile.
Pros are a huge improvement of UX of connecting both within and from outside of the app (e.g., link can be resolved even before creating chat profile, as part of the onboarding).
## Protocol
To support short links, the SMP servers would provide a simple key-value store enabled by three additional commands: `WRT`, `CLR` and `READ`
`WRT` command is used to store and to update values in the store. The size of the value is limited by the same size as sent messages (or, possibly, smaller - as connection information size used in confirmation messages) - the clients would use this fixed size irrespective of the content. `WRT` command will be sent with the data blob ID in the transaction entityId field, public authorization key used to authorize `WRT` and `CLR` commands (subsequent WRT commands to the existing key must use the same key), and the data blob.
`CLR` command must use with the same entity ID and must be authorized by the same key.
`READ` command must use the ID which hash would be equal of the ID used to create the data blob, and this ID would also be used as public authorization
## Algorithm to store and to retrieve data blob.
**Store data blob**
- the data blob owner generates X25519 key pair: `(k, pk)`.
- private key `pk` will be included in the short link shared with the other party (only base64url encoded key bytes, not X509 encoding).
- `HKDF(pk)` will be used to encrypt the link data with secret_box before storing it on the server.
- the hash of public key `sha256(k)` will be used as ID by the owner to store and to remove the data blob (`WRT` and `CLR` commands).
**Retrieve data blob**
- the sender uses the public key `k` derived from the private key `pk` included in the link as entity ID to retrieve data blob (the server will compute the ID used by the owner as `sha256(k)` and will be able to look it up). This provides the quality that the traffic of the parties has no shared IDs inside TLS. It also means that unlike message queue creation, the ID to retrieve the blob was never sent to the blob creator, and also is not known to the server in advance (the second part is only an observation, in itself it does not increase security, as server has access to an encrypted blob anyway).
- note that the sender does not authorize the request to retrieve the blob, as it would not increase security unless a different key is used to authorize, and adding a key would increase link size.
- server session keys with the sender will be `(sk, spk)`, where `sk` is public key shared with the sender during session handshake, and `spk` is the private key known only to the server.
- this public key `k` will also be combined with server session key `spk` using `dh(k, spk)` to encrypt the response, so that there is no ciphertext in common in sent and received traffic for these blobs. Correlation ID will be used as a nonce for this encryption.
- having received the blob, the client can now decrypt it using secret_box with `HKDF(pk)`.
Using the same key as ID for the request, and also to additionally encrypt the response allows to use a single key in the link, without increasing the link size.
## Threat model
**Compromised SMP server**
can:
- delete link data.
- hide link selectively from some requests.
cannot:
- undetectably replace link data.
- access unencrypted link data, whether it was or was not accessed by the accepting party.
- observe IP addresses of the users accessing link data.
**Passive observer who observed short link**:
can:
- access original unencrypted link data
cannot:
- replace or delete the link data

81
rfcs/2024-09-05-queue-storage.md Normal file
View File

@@ -0,0 +1,81 @@
# Storage considerations for SMP queues
See [Short invitation links](./2024-06-21-short-links.md).
## Problem
1) queue records are created permanently, until the clients delete them.
2) clients only delete queue records based on some user action, pending connections do not expire.
While part 2 should be improved in the client, indefinite storage of queue records becomes a much bigger issue if each of them would result in a permanent storage of 4-16kb blob in server memory, without server-side expiration for short invitation links.
## Possible solutions
1) Add some queue timestamp, e.g. queue creation date, to expire unsecured queues after say 3 weeks.
The problem with this approach is that contact addresses are also unsecured queues, and they should not be expired.
We could set really large expiration time, and require that clients "update" the unsecured queues they need at least every 1-2 years, but it would not solve the problem of storing a large number of blobs in the server memory for unused/abandoned 1-time invitations.
2) Do not store blobs in memory / append-only log, and instead use something like RocksDB. While it may be a correct long term solution, it may be not expedient enough at the current POC stage for this feature. Also, the lack of expiration is wrong in any case and would indefinitely grow server storage.
3) Add flag allowing the server to differentiate permanent queues used as contact addresses, also using different blob sizes for them. In this case, messaging queues will be expired if not secured after 3 weeks, and contact address queues would be expired if not "updated" by the owner within 2 years.
Probably all three solutions need to be used, to avoid creating a non-expiring blob storage in memory, as in case too many of such blobs are created it would not be possible to differentiate between real users and resource exhaustion attacks, and unlike with messages, they won't be expiring too.
Servers already can differentiate messaging queues and contact address queues, if they want to:
- with the old 4-message handshake, the confirmation message on a normal queue was different, and also KEY command was eventually used.
- with the fast 2-message handshake, while the confirmation message has the same syntax, and the differences are inside encrypted envelope, the client still uses SKEY command.
- in both cases, the usual messaging queues are secured, and contact addresses are not, so this difference is visible in the storage as well (although it is not easy to differentiate between abandoned 1-time invitations and contact addresses).
Differentiating these queues can also allow different message retention times - e.g., the queues for contact addresses could have bigger size, but have lower message retention time.
## Proposed solution
1. Add queue updated_at date into queue records. While it adds some metadata, it seems necessary to manage retention and quality of service. It will not include exact time, only date, and the time of creation will be replaced by the time of any update - queue secured, a message is sent, or queue owner subscribes to the queue. To avoid the need to update store log on every message this information can be appended to store log on server termination. Or given that only one update per day is needed it may be ok to make these updates as they happen (temporarily making the sequence and time of these events available in storage).
2. Add flag to indicate the queue usage - messaging queue or queue for contact address connection requests. This would result in different queue size and different retention policy for queue and its messages. We already have "sender can secure flag" which is, effectively, this flag - contact address queues are never secured. So this does not increase stored metadata in any way.
## Possible changes to short links
This is a design considerations and a concept, not a design yet.
Instead of implementing a generic blob storage that can be used as an attack vector, and adds additional failure point (another server storing blob that is necessary to connect to the queue on the current server), but instead adds an extended queue information blobs, most of which could be dropped without the loss of connectivity, so that the attack can be mitigated by deleting these blobs without users losing the ability to connect, as long as the queue and minimal extended information is retained.
So, to make the connection there need to be these elements:
- queue server and queue ID - mandatory part, that can be included in short link
- SMP key - mandatory part for all queues. We are considering initializing ratchets earlier for contact addresses, and include ratchet keys and pre-keys into queue data as well, but it is out of scope here.
- Ratchet keys - mandatory part for 1-time invitation that won't fit in short link.
- PQ key - optional part that can be stored with addresses if ratchet keys are added and with 1-time invitations.
- App blobs - chat preferences for 1-time invitation links and profile information for contact addresses.
So rather that storing one blob with a large address inside it, not associated with the queue, increasing probability of failure and reducing our ability to mitigate resource exhaustion, we could store extended blobs associated with the queues.
Also, we need the address shared with the sender (party accepting the connection) to be short. We could use a similar approach that was proposed for data blobs, using a single random seed per queues to derive multiple keys and IDs from it. For example:
1. The queue owner:
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now sent in NEW command.
- generates queue recipient ID (this ID can still be server-generated).
- generates X25519 key pair `(k, pk)` to use with the accepting party.
- derives from `k`:
- sender ID.
- symmetric key for authenticated encryption of blobs.
- `k` will be used as short link.
2. All other data from the invitation can be included in queue creation request and be associated with the queue as 1-3 blobs with different priority:
- ratchet keys - it will have a small size, so only this blob cannot be removed, while other blobs can be removed in case of resource exhaustion.
- PQ keys - optional blob.
- conversation preferences and profile - can be removed depending on creation time, e.g. all new blobs can be removed.
The algorithm used to derive key and ID from `k` needs to be cryptographically secure, e.g. it could be some KDF or ChaCha DRG initialized with `k` as seed, TBC.
So, coupling blob storage with messaging queues has these pros/cons:
Cons:
- no additional layer of privacy - the server used for connection is visible in the link, even after the blobs are removed from the server.
Pros:
- no additional point of failure in the connection process - the same server will be used to retrieve necessary blobs as for connection.
- queue blobs of messaging blobs will be automatically removed once the queue is secured or expired, without additional request from the recipient - reducing the storage and the time these blobs are available.
- queue blobs for contact addresses will be structured and some of the large blobs can be removed in case of resource exhaustion attack (and recreated by the client if needed), with the only downside that PQ handshake will be postponed (which is the case now) and profile will not be available at a point of connection.

80
rfcs/2024-09-09-smp-blobs.md Normal file
View File

@@ -0,0 +1,80 @@
# Blob extensions for SMP queues
Evolution of the design for short links, see [here](./2024-06-21-short-links.md) and [here](./2024-09-05-queue-storage.md).
## Problems
Allow storing extended information with SMP queues to improve UX and security of making connections:
- short invitation links and contact addresses.
- PQ encryption from the first message.
- present user profile with chat preferences and welcome message when the public address link is scanned.
## Design
1. Queue creation/update date is already added to server persistence, allowing to expire queues and blobs, depending on their usage.
2. Add "queue type" metadata to NEW command to indicate whether messaging queue is used as public address or as messaging queue (see previous docs on why it doesn't change threat model). While at the moment it would match sndSecure flag there may be future scenarios when they diverge. Initially only "invitation" and "contact" types will be supported.
3. Prohibit sndSecure flag for "contact" queues, prohibit securing contact queues.
4. Add "queue blobs" to NEW command:
- blob0: ratchetKeys up to N0 bytes - priority 0, can't be removed by the server, only in "invitation"
- blob1: PQ key up to N1 bytes - priority 1, can be removed by the server, only used in "invitation"
- blob2: Application data up to N2 bytes - priority 2, can be removed by the server.
5. Add linkId to NEW command
6. linkId and blobs will be removed when queue is secured.
7. Add recipient command to remove/upsert blob2 for contact queues.
8. Add sender command to retrieve blobs.
## Protocol
### Creating a queue:
The queue owner:
- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now. `sk` and `dhk` will be sent in NEW command.
- generates X25519 key pair `(k, pk)` to use with the accepting party to encrypt queue messages.
- derives from `k` using HKDF:
- symmetric key `bk` for authenticated encryption of blobs.
- `linkId`, will be sent in NEW command.
- `k` will be used as short link.
- sends NEW command.
NEW command syntax:
```abnf
create = %s"NEW " linkId queueType recipientAuthPublicKey recipientDhPublicKey
basicAuth subscribe sndSecure [ "0" blob0 ] [ "1" blob1 ] [ "2" blob2 ]
queueType = %s"I" / %s "C" ; new parameter
linkId = length *OCTET ; new parameter,
; can be empty in which case blobs won't be allowed
blob0 = word16 *OCTET ; new parameter, encrypted ratchet keys,
; including nonce and auth tag
blob1 = word16 *OCTET ; new parameter, encrypted PQ key
blob2 = word16 *OCTET ; new parameter, encrypted application data
```
SET - command to update queue blobs (recipientId is used as entity ID):
```abnf
set = %s"SET " linkId [ "2" blob2 ] ; passing empty blob removes it
linkId ; updated (or the same) linkId, can be empty to remove blobs
; allows to change the address without removing the queue / changing blobs
; (e.g., to avoid losing the messages).
```
### Sending messages to the queue
GET - command to get queue blobs (linkId is used as entity ID):
```abnf
get = %s"GET"
```
Response to GET:
```abnf
blobs = %s"BLOB" senderId [ "0" blob0 ] [ "1" blob1 ] [ "2" blob2 ]
```
As blobs are retrieved using a separate linkId, once blobs are removed it will be impossible to find senderId from short link - it is a threat model improvement. Once server storage is compacted, it will be impossible to find queue related to the link even with the access to server data (unless server preserves the data).
### Possible privacy improvement
We could only allow unauthorized GET and authorized SET commands for long-term "contact" queues, and return BLOB in response to SKEY (or require that GET is authorized) - so that only the person who secures the queue will get access to data blobs. This way it ensures that the parties transmitting the invitation links cannot retrieve their content without the sender noticing it.

26
rfcs/2024-09-10-private-rendezvous.md Normal file
View File

@@ -0,0 +1,26 @@
# Private rendezvous protocol
## Problem
Our current handshake protocol is open to this attack: whoever observes the link exchange, knows on which server connection is being made, and if the traffic on this server is observed, then it can confirm communication between parties. Further, even with the [last proposal](./2024-09-09-smp-blobs.md#possible-privacy-improvement), having real-time access to the server data allows to establish the exact messaging queue that is used to send messages.
## Solution
We could make the initial link exchange more private by making it harder for any observer to discover which server will be used for messaging by hiding this information from the server that hosts the initial link.
Preliminary, the protocol could be the following:
1. Connection initiator stores 224-256 bytes of encrypted connection link on a rendezvous server (link contains server host and linkId on another messaging server, not a rendezvous one).
2. Rendezvous server adds these links to buckets, up to 64 links per bucket. Bucket ID is the timestamp when the bucket was created + a sequential bucket number, in case more than one bucket is created per second.
3. The server responds to the link creator with a bucket ID where this link was added. That bucket ID is its timestamp + a number prevents server "fingerprinting" clients and using say one bucket for each client. If timestamp is different or a bucket number within this timestamp is too large, the client can refuse to use it, depending on the client settings.
4. The initiating party will pass to the accepting party the rendezvous server host, the hash of this bucket ID (bucket link) and the passphrase to derive the key from. The initiating party has an option to pass a link and passphrase via two channels - in which case the link will only contain the bucket ID.
5. The accepting party would then request the bucket via its ID hash (the server would store hashes to be able to look up - hash is used to prevent showing time in the link) and attempt to decrypt all contained links using the provided key.
The accepting party then will continue the connection via the decrypted link.
This obviously does not protect accepting party from the initiating party, if it can choose rendezvous server it controls. It also does not protect from the malicious rendezvous server that would collaborate with link observers. I think reunion doesn’t protect from it too.
But it does protect connection from whoever observes the link, particularly if this link only contains the bucket and the key is passed separately, via some other channel.