Merge branch 'master' into master-android

This commit is contained in:
Evgeny Poberezkin
2024-05-02 16:48:46 +01:00
9 changed files with 176 additions and 44 deletions

View File

@@ -453,7 +453,11 @@ const processCommand = (function () {
// For opus (where encodedFrame.type is not set) this is the TOC byte from
// https://tools.ietf.org/html/rfc6716#section-3.1
var _a;
const capabilities = RTCRtpSender.getCapabilities("video");
// Using RTCRtpReceiver instead of RTCRtpSender, see these lines:
// - if (!is_recv_codec && !is_send_codec) {
// + if (!is_recv_codec) {
// https://webrtc.googlesource.com/src.git/+/db2f52ba88cf9f98211df2dabb3f8aca9251c4a2%5E%21/
const capabilities = RTCRtpReceiver.getCapabilities("video");
if (capabilities) {
const { codecs } = capabilities;
const selectedCodecIndex = codecs.findIndex((c) => c.mimeType === "video/VP8");
@@ -464,7 +468,13 @@ const processCommand = (function () {
// Firefox doesn't have this function implemented:
// https://bugzilla.mozilla.org/show_bug.cgi?id=1396922
if (((_a = t.sender.track) === null || _a === void 0 ? void 0 : _a.kind) === "video" && t.setCodecPreferences) {
t.setCodecPreferences(codecs);
try {
t.setCodecPreferences(codecs);
}
catch (error) {
// Shouldn't be here but in case something goes wrong, it will allow to make a call with auto-selected codecs
console.log("Failed to set codec preferences, trying without any preferences: " + error);
}
}
}
}

View File

@@ -12,7 +12,7 @@ constraints: zip +disable-bzip2 +disable-zstd
source-repository-package
type: git
location: https://github.com/simplex-chat/simplexmq.git
tag: 935d2e25df80eed7761c48a01529b6c0da0d3baa
tag: 8d8010a62aef2241fec3876fcfe57d51456b2bc0
source-repository-package
type: git

View File

@@ -0,0 +1,68 @@
# Commercial model for SimpleX communication network
## Problem
SimpleX two-tier network design provides a _potential_ for a much higher degree of decentralization and privacy than a p2p network can achieve even theoretically. This is a very strong statement, and its formal proof is out of scope of this document, please see the old comparison of SimpleX network and p2p network designs [here](../SIMPLEX.md#comparison-with-p2p-messaging-protocols).
The main downside of most, if not all, p2p networks is the lack of (or very limited) asynchronous message delivery that is critically important both for the network usability and for privacy and protection against traffic correlation attacks.
But while two-tier network design has a _potential_ for higher privacy, this potential is hard to realize as it does not have an in-built mechanism for network operator incentives. All SimpleX network relays preset in the app are operated by SimpleX Chat Ltd., and while there are probably over 1000 self-hosted and community-hosted messaging relays - not having a single register of such relays is important for true decentralization, so we cannot know the exact number - it is probably a valid argument that a substantial part of SimpleX network traffic is provided by preset relays. Again, while we don't know the exact share, it is probably not much less than 40-50% and probably not much more than 65-75% - which is a very large share in any case for a single entity.
For SimpleX network to achieve the level of decentralization that is designed, the applications have to be able to offer multiple network operators via the app who have commercial incentives to operate these relays.
Traditional commercial models for consumer Internet products rely on some of these ideas:
1. Offer service for free, sell users' data. We obviously had enough of such Internet and can't wait to see it implode, lose all users, and become illegal.
2. Create a cryptocurrency or issue cryptocurrency tokens, sell them for real money, and use this money to fund service operation. While many people believe that this is the future of the Web, calling it Web3, we see it as a technological dead end, that although it provides a great platform for speculation and a playground to test such ideas as smart contracts and consensus algorithms, all of which can be used outside of the context of cryptocurrency blockchains, is not a sustainable technological foundation for a general purpose communication or information management system due to its inherent regulatory risks and distorted commercial incentives. Moxie Marlinspike wrote [a good critique of Web3](https://moxie.org/2022/01/07/web3-first-impressions.html).
3. Freemium models where some users pay for the service and, effectively, sponsor the users of the free tier. This model is free from the downsides of the previous two options, and there are many commercial services operated on this model, but this model is suboptimal for privacy in the worst case, and in the best case it still does not achieve decentralization, as whoever charges the money (the app provider) should also provide the infrastructure.
In the mentioned critique of Web3 Moxie wrote: _We should accept the premise that people will not run their own servers by designing systems that can distribute trust without having to distribute infrastructure_.
This statement is interesting, as it contains the correct premise - that most people do not want to and won't run their own servers - but it reaches an incorrect and limited conclusion, that the only way to provide value is by figuring out how to decentralize the trust without decentralizing infrastructure. I completely disagree that this conclusion is the only one possible, and the offered solution actually offers a way for extreme infrastructure decentralization - when not just users are distributed across providers, as happens with federated designs, but each conversation between two users is supported by infrastructure of 4-6 different independent operators - and also provides commercial incentives for these operators.
## Solution design requirements
So, we want to find a solution that will:
1. Offer extreme decentralization, without any kind of central authority (unlike Tor that has central authority, and unlike p2p networks that have a single addressing space) or any kind of centralized state (unlike cryptocurrency blockchains that have a centralized single state that the whole network should reach consensus about).
2. Offer low barrier for entry for infrastructure operators.
3. Offer extreme data and infrastructure portability, when infrastructure operators offer standardized primitives used by client applications - such as messaging queues and file chunks in SimpleX network. Migrating from one operator to another should not be just possible or simple, it should happen continuously and automatically, all the time, without any user action, when files and conversations move from one set of operators to another, similarly to how data moves around on SSD drives - to balance the load on the whole network, to provide reliability and redundancy, and to ensure that infrastructure operators have zero control of users and their data, and have very limited knowledge of users activity. This limited knowledge both achieves users' privacy and reduces the legal responsibility of infrastructure operators to the level of network operators.
4. Offer commercially profitable model for infrastructure operators.
If these requirements are satisfied it would achieve a radical shift of control from infrastructure operators of today (that is achieved via SaaS model, when infrastructure operators are also software vendors, and software is provided as a service, which seems to be the root cause of corruption of Internet services - the process that Cory Doctorow refers to as [enshittification](https://www.youtube.com/watch?v=q118B_QdP2k)) to the software users, who use client software to directly consume low level infrastructure primitives of commoditized infrastructure operators that have zero control of how these primitives are used, other than pricing, that can be determined in a competitive process (e.g., via the real-time reverse auction).
## Solution concept
There are three types of participants in the network:
- software vendors (e.g., SimpleX Chat ltd.).
- infrastructure operators - commercial entities that have agreement with software vendors - they run server software provided by software vendors.
- users - they run client software.
These roles can obviously overlap, but strategically it is better if they don't, e.g. we would benefit from not operating infrastructure, and users would also achieve better metadata privacy by not running the servers.
To use SimpleX network client software needs to provision simple infrastructure resources - such as, create a messaging queue to receive the messages, create or use a session from a sending proxy to a particular destination relay, or to upload a file chunk that will be stored for a defined number of days. These resources are very cheap to provide, their price could be a very tiny fraction of a cent.
For the users' client software to be able to provision these resources, the software vendor will issue infrastructure certificates to the users. Some number of them can be provided for free, a larger number can be sold for a monetary payment (can be in-app purchase, or cryptocurrency payment, or any other process). Software vendor will keep a private record of issued certificates. Technically, certificates can be usual cryptographic certificates that sign a public key presented by the client (while the client holds the private key).
These certificates cannot be used as a cryptocurrency, as there is no public record of all issued certificates, and they can only be "spent" once with the infrastructure operator that has a commercial agreement with a software vendor.
When the client wants to provision infrastructure resource it signs the request using the private key (a public counterpart of which was signed by the vendor's root certificate) and present this signed request together with the client's certificate to infrastructure operator. Operator validates the signature and certificate using vendor's root certificate and immediately provisions the resource.
Operator then within a limited time presents its own signed request for payment to the software vendor - it would include the original request, without specifying which resource was provisioned, but only confirming that it was to this vendor. Software vendor can either confirm the acceptance and void the certificate or inform the vendor that this certificate was previously used (double spend), in which case the operator will stop provisioning the resource.
Pros:
- this design decouples payments from resource allocation, providing better privacy, and decentralizes the infrastructure.
- this design achieves extreme provider portability and lack of control of user and user data.
Cons:
- this design creates a strong dependence on software vendor's payment infrastructure availability - even though there can be many software vendors using this approach in a compatible way, it still creates a strong dependence of client software functioning on a single vendor.
- as described, this design allows software vendor to correlate payments of a given software user to specific infrastructure operators.
Problem 2 can be solved by using some sort of [zero-knowledge proofs](https://en.wikipedia.org/wiki/Non-interactive_zero-knowledge_proof), where infrastructure operator can prove to software vendor that 1) they received a valid request 2) software vendor can also prove that the same certificate was not presented before, but cannot determine which certificate it was.
Problem 1 can possibly be solved by delegating the right to issue a limited number of certificate to infrastructure operators, and making all records in the private blockchain accessible and writable by the operators using the agreements with the vendor that would also be visible on this chain, so that the operators cannot issue more certificates than agreed. In this case the payments will be made not by the software vendor to the operators but by the operators to the vendor, by compensating the buy/sell price difference.
This is a concept of design, rather than the actual design, and the details of cryptographic primitives and consensus algorithms for this chain are out of scope. It is important that this design limits the number of operations with each certificate to three:
1. certificate issued to the user, based on the rules agreed with the software vendor.
2. certificate is used as a micro-payment for infrastructure resource (it cannot be transferred to any other user without risks of double spend).
3. certificate can only be voided by software vendor (or delegate who issued it), so it cannot be used directly as a payment.
If we see cryptocurrency as similar to money, with similar regulations, this cryptographic primitive is close to gift cards, that have zero monetary value and can be only exchanged to a specific resource/service, thus avoiding the usual regulatory risks, and also avoiding speculative hype that could decouple the value of the certificates from the price of the infrastructure.

View File

@@ -657,7 +657,11 @@ const processCommand = (function () {
// For opus (where encodedFrame.type is not set) this is the TOC byte from
// https://tools.ietf.org/html/rfc6716#section-3.1
const capabilities = RTCRtpSender.getCapabilities("video")
// Using RTCRtpReceiver instead of RTCRtpSender, see these lines:
// - if (!is_recv_codec && !is_send_codec) {
// + if (!is_recv_codec) {
// https://webrtc.googlesource.com/src.git/+/db2f52ba88cf9f98211df2dabb3f8aca9251c4a2%5E%21/
const capabilities = RTCRtpReceiver.getCapabilities("video")
if (capabilities) {
const {codecs} = capabilities
const selectedCodecIndex = codecs.findIndex((c) => c.mimeType === "video/VP8")
@@ -668,7 +672,12 @@ const processCommand = (function () {
// Firefox doesn't have this function implemented:
// https://bugzilla.mozilla.org/show_bug.cgi?id=1396922
if (t.sender.track?.kind === "video" && t.setCodecPreferences) {
t.setCodecPreferences(codecs)
try {
t.setCodecPreferences(codecs)
} catch (error) {
// Shouldn't be here but in case something goes wrong, it will allow to make a call with auto-selected codecs
console.log("Failed to set codec preferences, trying without any preferences: " + error)
}
}
}
}

View File

@@ -1,5 +1,5 @@
{
"https://github.com/simplex-chat/simplexmq.git"."935d2e25df80eed7761c48a01529b6c0da0d3baa" = "1gyis2zvxm6352x3myaxdifr9sy4fkhygwqmzgxvn669gmf4gx2n";
"https://github.com/simplex-chat/simplexmq.git"."8d8010a62aef2241fec3876fcfe57d51456b2bc0" = "0x7fq33c0x7i9jjp42la3zkha1wk6s3bv7dkz9z39a02s9rfkfla";
"https://github.com/simplex-chat/hs-socks.git"."a30cc7a79a08d8108316094f8f2f82a0c5e1ac51" = "0yasvnr7g91k76mjkamvzab2kvlb1g5pspjyjn2fr6v83swjhj38";
"https://github.com/simplex-chat/direct-sqlcipher.git"."f814ee68b16a9447fbb467ccc8f29bdd3546bfd9" = "1ql13f4kfwkbaq7nygkxgw84213i0zm7c1a8hwvramayxl38dq5d";
"https://github.com/simplex-chat/sqlcipher-simple.git"."a46bd361a19376c5211f1058908fc0ae6bf42446" = "1z0r78d8f0812kxbgsm735qf6xx8lvaz27k1a0b4a2m0sshpd5gl";

View File

@@ -2932,7 +2932,7 @@ callTimed ct aciContent =
case aciContentCallStatus aciContent of
Just callStatus
| callComplete callStatus -> do
contactCITimed ct
contactCITimed ct
_ -> pure Nothing
where
aciContentCallStatus :: ACIContent -> Maybe CICallStatus
@@ -3577,14 +3577,19 @@ processAgentMessageNoConn = \case
processAgentMsgSndFile :: ACorrId -> SndFileId -> ACommand 'Agent 'AESndFile -> CM ()
processAgentMsgSndFile _corrId aFileId msg = do
fileId <- withStore (`getXFTPSndFileDBId` AgentSndFileId aFileId)
withFileLock "processAgentMsgSndFile" fileId $
(cRef_, fileId) <- withStore (`getXFTPSndFileDBIds` AgentSndFileId aFileId)
withEntityLock_ cRef_ $ withFileLock "processAgentMsgSndFile" fileId $
withStore' (`getUserByASndFileId` AgentSndFileId aFileId) >>= \case
Just user -> process user fileId `catchChatError` (toView . CRChatError (Just user))
_ -> do
lift $ withAgent' (`xftpDeleteSndFileInternal` aFileId)
throwChatError $ CENoSndFileUser $ AgentSndFileId aFileId
where
withEntityLock_ :: Maybe ChatRef -> CM a -> CM a
withEntityLock_ cRef_ = case cRef_ of
Just (ChatRef CTDirect contactId) -> withContactLock "processAgentMsgSndFile" contactId
Just (ChatRef CTGroup groupId) -> withGroupLock "processAgentMsgSndFile" groupId
_ -> id
process :: User -> FileTransferId -> CM ()
process user fileId = do
(ft@FileTransferMeta {xftpRedirectFor, cancelled}, sfts) <- withStore $ \db -> getSndFileTransfer db user fileId
@@ -3699,14 +3704,19 @@ splitFileDescr rfdText = do
processAgentMsgRcvFile :: ACorrId -> RcvFileId -> ACommand 'Agent 'AERcvFile -> CM ()
processAgentMsgRcvFile _corrId aFileId msg = do
fileId <- withStore (`getXFTPRcvFileDBId` AgentRcvFileId aFileId)
withFileLock "processAgentMsgRcvFile" fileId $
(cRef_, fileId) <- withStore (`getXFTPRcvFileDBIds` AgentRcvFileId aFileId)
withEntityLock_ cRef_ $ withFileLock "processAgentMsgRcvFile" fileId $
withStore' (`getUserByARcvFileId` AgentRcvFileId aFileId) >>= \case
Just user -> process user fileId `catchChatError` (toView . CRChatError (Just user))
_ -> do
lift $ withAgent' (`xftpDeleteRcvFile` aFileId)
throwChatError $ CENoRcvFileUser $ AgentRcvFileId aFileId
where
withEntityLock_ :: Maybe ChatRef -> CM a -> CM a
withEntityLock_ cRef_ = case cRef_ of
Just (ChatRef CTDirect contactId) -> withContactLock "processAgentMsgRcvFile" contactId
Just (ChatRef CTGroup groupId) -> withGroupLock "processAgentMsgRcvFile" groupId
_ -> id
process :: User -> FileTransferId -> CM ()
process user fileId = do
ft <- withStore $ \db -> getRcvFileTransfer db user fileId
@@ -4259,12 +4269,8 @@ processAgentMessageConn vr user@User {userId} corrId agentConnId agentMessage =
Right (ACMsg _ chatMsg) ->
processEvent chatMsg `catchChatError` \e -> toView $ CRChatError (Just user) e
Left e -> toView $ CRChatError (Just user) (ChatError . CEException $ "error parsing chat message: " <> e)
forwardMsg_ `catchChatError` \_ -> pure ()
checkSendRcpt $ rights aChatMsgs
-- currently only a single message is forwarded
let GroupMember {memberRole = membershipMemRole} = membership
when (membershipMemRole >= GRAdmin && not (blockedByAdmin m)) $ case aChatMsgs of
[Right (ACMsg _ chatMsg)] -> forwardMsg_ chatMsg
_ -> pure ()
where
aChatMsgs = parseChatMessages msgBody
brokerTs = metaBrokerTs msgMeta
@@ -4312,22 +4318,27 @@ processAgentMessageConn vr user@User {userId} corrId agentConnId agentMessage =
where
aChatMsgHasReceipt (ACMsg _ ChatMessage {chatMsgEvent}) =
hasDeliveryReceipt (toCMEventTag chatMsgEvent)
forwardMsg_ :: MsgEncodingI e => ChatMessage e -> CM ()
forwardMsg_ chatMsg =
forM_ (forwardedGroupMsg chatMsg) $ \chatMsg' -> do
ChatConfig {highlyAvailable} <- asks config
-- members introduced to this invited member
introducedMembers <-
if memberCategory m == GCInviteeMember
then withStore' $ \db -> getForwardIntroducedMembers db vr user m highlyAvailable
else pure []
-- invited members to which this member was introduced
invitedMembers <- withStore' $ \db -> getForwardInvitedMembers db vr user m highlyAvailable
let GroupMember {memberId} = m
ms = forwardedToGroupMembers (introducedMembers <> invitedMembers) chatMsg'
msg = XGrpMsgForward memberId chatMsg' brokerTs
unless (null ms) . void $
sendGroupMessage' user gInfo ms msg
forwardMsg_ :: CM ()
forwardMsg_ = do
let GroupMember {memberRole = membershipMemRole} = membership
when (membershipMemRole >= GRAdmin && not (blockedByAdmin m)) $ case aChatMsgs of
-- currently only a single message is forwarded
[Right (ACMsg _ chatMsg)] ->
forM_ (forwardedGroupMsg chatMsg) $ \chatMsg' -> do
ChatConfig {highlyAvailable} <- asks config
-- members introduced to this invited member
introducedMembers <-
if memberCategory m == GCInviteeMember
then withStore' $ \db -> getForwardIntroducedMembers db vr user m highlyAvailable
else pure []
-- invited members to which this member was introduced
invitedMembers <- withStore' $ \db -> getForwardInvitedMembers db vr user m highlyAvailable
let GroupMember {memberId} = m
ms = forwardedToGroupMembers (introducedMembers <> invitedMembers) chatMsg'
msg = XGrpMsgForward memberId chatMsg' brokerTs
unless (null ms) . void $
sendGroupMessage' user gInfo ms msg
_ -> pure ()
RCVD msgMeta msgRcpt ->
withAckMessage' agentConnId msgMeta $
groupMsgReceived gInfo m conn msgMeta msgRcpt

View File

@@ -29,8 +29,8 @@ module Simplex.Chat.Store.Files
createExtraSndFTDescrs,
updateSndFTDeliveryXFTP,
setSndFTAgentDeleted,
getXFTPSndFileDBId,
getXFTPRcvFileDBId,
getXFTPSndFileDBIds,
getXFTPRcvFileDBIds,
updateFileCancelled,
updateCIFileStatus,
getSharedMsgIdByFileId,
@@ -109,7 +109,7 @@ import Simplex.Chat.Store.Shared
import Simplex.Chat.Types
import Simplex.Chat.Util (week)
import Simplex.Messaging.Agent.Protocol (AgentMsgId, ConnId, UserId)
import Simplex.Messaging.Agent.Store.SQLite (firstRow, maybeFirstRow)
import Simplex.Messaging.Agent.Store.SQLite (firstRow, firstRow', maybeFirstRow)
import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Crypto.File (CryptoFile (..), CryptoFileArgs (..))
@@ -336,15 +336,37 @@ setSndFTAgentDeleted db User {userId} fileId = do
"UPDATE files SET agent_snd_file_deleted = 1, updated_at = ? WHERE user_id = ? AND file_id = ?"
(currentTs, userId, fileId)
getXFTPSndFileDBId :: DB.Connection -> AgentSndFileId -> ExceptT StoreError IO FileTransferId
getXFTPSndFileDBId db aSndFileId =
ExceptT . firstRow fromOnly (SESndFileNotFoundXFTP aSndFileId) $
DB.query db "SELECT file_id FROM files WHERE agent_snd_file_id = ?" (Only aSndFileId)
getXFTPSndFileDBIds :: DB.Connection -> AgentSndFileId -> ExceptT StoreError IO (Maybe ChatRef, FileTransferId)
getXFTPSndFileDBIds db aSndFileId =
ExceptT . firstRow' toFileRef (SESndFileNotFoundXFTP aSndFileId) $
DB.query
db
[sql|
SELECT file_id, contact_id, group_id, note_folder_id
FROM files
WHERE agent_snd_file_id = ?
|]
(Only aSndFileId)
getXFTPRcvFileDBId :: DB.Connection -> AgentRcvFileId -> ExceptT StoreError IO FileTransferId
getXFTPRcvFileDBId db aRcvFileId =
ExceptT . firstRow fromOnly (SERcvFileNotFoundXFTP aRcvFileId) $
DB.query db "SELECT file_id FROM rcv_files WHERE agent_rcv_file_id = ?" (Only aRcvFileId)
getXFTPRcvFileDBIds :: DB.Connection -> AgentRcvFileId -> ExceptT StoreError IO (Maybe ChatRef, FileTransferId)
getXFTPRcvFileDBIds db aRcvFileId =
ExceptT . firstRow' toFileRef (SERcvFileNotFoundXFTP aRcvFileId) $
DB.query
db
[sql|
SELECT rf.file_id, f.contact_id, f.group_id, f.note_folder_id
FROM rcv_files rf
JOIN files f ON f.file_id = rf.file_id
WHERE rf.agent_rcv_file_id = ?
|]
(Only aRcvFileId)
toFileRef :: (FileTransferId, Maybe Int64, Maybe Int64, Maybe Int64) -> Either StoreError (Maybe ChatRef, FileTransferId)
toFileRef = \case
(fileId, Just contactId, Nothing, Nothing) -> Right (Just $ ChatRef CTDirect contactId, fileId)
(fileId, Nothing, Just groupId, Nothing) -> Right (Just $ ChatRef CTGroup groupId, fileId)
(fileId, Nothing, Nothing, Just folderId) -> Right (Just $ ChatRef CTLocal folderId, fileId)
(fileId, _, _, _) -> Right (Nothing, fileId)
updateFileCancelled :: MsgDirectionI d => DB.Connection -> User -> Int64 -> CIFileStatus d -> IO ()
updateFileCancelled db User {userId} fileId ciFileStatus = do

View File

@@ -146,7 +146,6 @@ import Simplex.Messaging.Agent.Store.SQLite (firstRow, firstRow', maybeFirstRow)
import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Crypto.File (CryptoFile (..), CryptoFileArgs (..))
import Simplex.Messaging.Crypto.Ratchet (PQSupport)
import Simplex.Messaging.Util (eitherToMaybe)
import UnliftIO.STM

View File

@@ -120,6 +120,7 @@ chatGroupTests = do
it "forward file (x.msg.file.descr)" testGroupMsgForwardFile
it "forward role change (x.grp.mem.role)" testGroupMsgForwardChangeRole
it "forward new member announcement (x.grp.mem.new)" testGroupMsgForwardNewMember
it "forward member leaving (x.grp.leave)" testGroupMsgForwardLeave
describe "group history" $ do
it "text messages" testGroupHistory
it "history is sent when joining via group link" testGroupHistoryGroupLink
@@ -4437,6 +4438,18 @@ testGroupMsgForwardNewMember =
"dan (Daniel): member"
]
testGroupMsgForwardLeave :: HasCallStack => FilePath -> IO ()
testGroupMsgForwardLeave =
testChat3 aliceProfile bobProfile cathProfile $
\alice bob cath -> do
setupGroupForwarding3 "team" alice bob cath
bob ##> "/leave #team"
bob <## "#team: you left the group"
bob <## "use /d #team to delete the group"
alice <## "#team: bob left the group"
cath <## "#team: bob left the group"
testGroupHistory :: HasCallStack => FilePath -> IO ()
testGroupHistory =
testChat3 aliceProfile bobProfile cathProfile $