mirror of
https://github.com/simplex-chat/simplexmq.git
synced 2026-04-27 12:55:16 +00:00
f6aca47604
* xftp: implementation of XFTP client as web page (rfc, low level functions) * protocol, file descriptions, more cryptogrpahy, handshake encoding, etc. * xftp server changes to support web slients: SNI-based certificate choice, CORS headers, OPTIONS request * web handshake * test for xftp web handshake * xftp-web client functions, fix transmission encoding * support description "redirect" in agent.ts and cross-platform compatibility tests (Haskell <> TypeScript) * rfc: web transport * client transport abstraction * browser environment * persistent client sessions * move rfcs * web page plan * improve plan * webpage implementation (not tested) * fix test * fix test 2 * fix test 3 * fixes and page test plan * allow sending xftp client hello after handshake - for web clients that dont know if established connection exists * page tests pass * concurrent and padded hellos in the server * update TS client to pad hellos * fix tests * preview:local * local preview over https * fixed https in the test page * web test cert fixtures * debug logging in web page and server * remove debug logging in server/browser, run preview xftp server via cabal run to ensure the latest code is used * debug logging for page sessions * add plan * improve error handling, handle browser reconnections/re-handshake * fix * debugging * opfs fallback * delete test screenshot * xftp CLI to support link * fix encoding for XFTPServerHandshake * support redirect file descriptions in xftp CLI receive * refactor CLI redirect * xftp-web: fixes and multi-server upload (#1714) * fix: await sodium.ready in crypto/keys.ts (+ digest.ts StateAddress cast) * multi-server parallel upload, remove pickRandomServer * fix worker message race: wait for ready signal before posting messages * suppress vite build warnings: emptyOutDir, externals, chunkSizeWarningLimit * fix Haskell web tests: use agent+server API, wrap server in array, suppress debug logs * remove dead APIs: un-export connectXFTP, delete closeXFTP * fix TypeScript errors in check:web (#1716) - client.ts: cast globalThis.process to any for browser tsconfig, suppress node:http2 import, use any for Buffer/chunks, cast fetch body - crypto.worker.ts: cast sha512_init() return to StateAddress * fix: serialize worker message processing to prevent OPFS handle race async onmessage allows interleaved execution at await points. When downloadFileRaw fetches chunks from multiple servers in parallel, concurrent handleDecryptAndStore calls both see downloadWriteHandle as null and race on createSyncAccessHandle for the same file, causing intermittent NoModificationAllowedError. Chain message handlers on a promise queue so each runs to completion before the next starts. * xftp-web: prepare for npm publishing (#1715) * prepare package.json for npm publishing Remove private flag, add description/license/repository/publishConfig, rename postinstall to pretest, add prepublishOnly, set files and main. * stable output filenames in production build * fix repository url format, expand files array * embeddable component: scoped CSS, dark mode, i18n, events, share - worker output to assets/ for single-directory deployment - scoped all CSS under #app, removed global resets - dark mode via .dark ancestor class - progress ring reads colors from CSS custom properties - i18n via window.__XFTP_I18N__ with t() helper - configurable mount element via data-xftp-app attribute - optional hashchange listener (data-no-hashchange) - completion events: xftp:upload-complete, xftp:download-complete - enhanced file-too-large error mentioning SimpleX app - native share button via navigator.share * deferred init and runtime server configuration - data-defer-init attribute skips auto-initialization - window.__XFTP_SERVERS__ overrides baked-in server list * use relative base path for relocatable build output * xftp-web: retry resets to default state, use innerHTML for errors * xftp-web: only enter download mode for valid XFTP URIs in hash * xftp-web: render UI before WASM is ready Move sodium.ready await after UI initialization so the upload/download interface appears instantly. WASM is only needed when user triggers an actual upload or download. Dispatch xftp:ready event once WASM loads. * xftp-web: CLS placeholder HTML and embedder CSS selectors Add placeholder HTML to index.html so the page renders a styled card before JS executes, preventing layout shift. Use a <template> element with an inline script to swap to the download placeholder when the URL hash indicates a file download. Auto-compute CSP SHA-256 hashes for inline scripts in the vite build plugin. Change all CSS selectors from #app to :is(#app, [data-xftp-app]) so styles apply when the widget is embedded with data-xftp-app attribute. * xftp-web: progress ring overhaul Rewrite progress ring with smooth lerp animation, green checkmark on completion, theme reactivity via MutationObserver, and per-phase color variables (encrypt/upload/download/decrypt). Show honest per-phase progress: each phase animates 0-100% independently with a ring color change between phases. Add decrypt progress callback from the web worker so the decryption phase tracks real chunk processing instead of showing an indeterminate spinner. Snap immediately on phase reset (0) and completion (1) to avoid lingering partial progress. Clean up animation and observers via destroy() in finally blocks. * xftp-web: single progress ring for upload, simplify ring color * xftp-web: single progress ring for download * feat(xftp-web): granular progress for encrypt/decrypt phases Add byte-level progress callbacks to encryptFile, decryptChunks, and sha512Streaming by processing data in 256KB segments. Worker reports fine-grained progress across all phases (encrypt+hash+write for upload, read+hash+decrypt for download). Progress ring gains fillTo method for smooth ease-out animation during minimum display delays. Encrypt/decrypt phases fill their weighted regions (0-15% and 85-99%) with real callbacks, with fillTo covering remaining time when work finishes under the 1s minimum for files >= 100KB. * rename package --------- Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com> --------- Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com> Co-authored-by: shum <github.shum@liber.li> Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
337 lines
11 KiB
Haskell
337 lines
11 KiB
Haskell
{-# LANGUAGE DataKinds #-}
|
|
{-# LANGUAGE DuplicateRecordFields #-}
|
|
{-# LANGUAGE LambdaCase #-}
|
|
{-# LANGUAGE MultiWayIf #-}
|
|
{-# LANGUAGE NamedFieldPuns #-}
|
|
{-# LANGUAGE OverloadedStrings #-}
|
|
{-# LANGUAGE PatternSynonyms #-}
|
|
{-# LANGUAGE ScopedTypeVariables #-}
|
|
{-# LANGUAGE TemplateHaskell #-}
|
|
|
|
module Simplex.FileTransfer.Transport
|
|
( supportedFileServerVRange,
|
|
authCmdsXFTPVersion,
|
|
blockedFilesXFTPVersion,
|
|
xftpClientHandshakeStub,
|
|
alpnSupportedXFTPhandshakes,
|
|
xftpALPNv1,
|
|
XFTPClientHandshake (..),
|
|
-- xftpClientHandshake,
|
|
XFTPServerHandshake (..),
|
|
-- xftpServerHandshake,
|
|
XFTPClientHello (..),
|
|
THandleXFTP,
|
|
THandleParamsXFTP,
|
|
VersionXFTP,
|
|
VersionRangeXFTP,
|
|
XFTPVersion,
|
|
pattern VersionXFTP,
|
|
XFTPErrorType (..),
|
|
XFTPRcvChunkSpec (..),
|
|
ReceiveFileError (..),
|
|
receiveFile,
|
|
sendEncFile,
|
|
receiveEncFile,
|
|
receiveSbFile,
|
|
)
|
|
where
|
|
|
|
import Control.Applicative (optional)
|
|
import qualified Control.Exception as E
|
|
import Control.Logger.Simple
|
|
import Control.Monad
|
|
import Control.Monad.Except
|
|
import Control.Monad.IO.Class
|
|
import Control.Monad.Trans.Except
|
|
import qualified Data.Aeson.TH as J
|
|
import qualified Data.Attoparsec.ByteString.Char8 as A
|
|
import Data.Bifunctor (first)
|
|
import qualified Data.ByteArray as BA
|
|
import Data.ByteString.Builder (Builder, byteString)
|
|
import Data.ByteString.Char8 (ByteString)
|
|
import qualified Data.ByteString.Char8 as B
|
|
import qualified Data.ByteString.Lazy.Char8 as LB
|
|
import Data.Functor (($>))
|
|
import Data.Word (Word16, Word32)
|
|
import Network.HTTP2.Client (HTTP2Error)
|
|
import qualified Simplex.Messaging.Crypto as C
|
|
import qualified Simplex.Messaging.Crypto.Lazy as LC
|
|
import Simplex.Messaging.Encoding
|
|
import Simplex.Messaging.Encoding.String
|
|
import Simplex.Messaging.Parsers
|
|
import Simplex.Messaging.Protocol (BlockingInfo, CommandError)
|
|
import Simplex.Messaging.Transport (ALPN, CertChainPubKey, ServiceCredentials, SessionId, THandle (..), THandleParams (..), TransportError (..), TransportPeer (..))
|
|
import Simplex.Messaging.Transport.HTTP2.File
|
|
import Simplex.Messaging.Util (bshow, tshow, (<$?>))
|
|
import Simplex.Messaging.Version
|
|
import Simplex.Messaging.Version.Internal
|
|
import System.IO (Handle, IOMode (..), withFile)
|
|
|
|
data XFTPRcvChunkSpec = XFTPRcvChunkSpec
|
|
{ filePath :: FilePath,
|
|
chunkSize :: Word32,
|
|
chunkDigest :: ByteString
|
|
}
|
|
deriving (Show)
|
|
|
|
data XFTPVersion
|
|
|
|
instance VersionScope XFTPVersion
|
|
|
|
type VersionXFTP = Version XFTPVersion
|
|
|
|
type VersionRangeXFTP = VersionRange XFTPVersion
|
|
|
|
pattern VersionXFTP :: Word16 -> VersionXFTP
|
|
pattern VersionXFTP v = Version v
|
|
|
|
type THandleXFTP c p = THandle XFTPVersion c p
|
|
type THandleParamsXFTP p = THandleParams XFTPVersion p
|
|
|
|
initialXFTPVersion :: VersionXFTP
|
|
initialXFTPVersion = VersionXFTP 1
|
|
|
|
authCmdsXFTPVersion :: VersionXFTP
|
|
authCmdsXFTPVersion = VersionXFTP 2
|
|
|
|
blockedFilesXFTPVersion :: VersionXFTP
|
|
blockedFilesXFTPVersion = VersionXFTP 3
|
|
|
|
currentXFTPVersion :: VersionXFTP
|
|
currentXFTPVersion = VersionXFTP 3
|
|
|
|
supportedFileServerVRange :: VersionRangeXFTP
|
|
supportedFileServerVRange = mkVersionRange initialXFTPVersion currentXFTPVersion
|
|
|
|
-- XFTP protocol does not use this handshake method
|
|
xftpClientHandshakeStub :: c 'TClient -> Maybe C.KeyPairX25519 -> C.KeyHash -> VersionRangeXFTP -> Bool -> Maybe (ServiceCredentials, C.KeyPairEd25519) -> ExceptT TransportError IO (THandle XFTPVersion c 'TClient)
|
|
xftpClientHandshakeStub _c _ks _keyHash _xftpVRange _proxyServer _serviceKeys = throwE TEVersion
|
|
|
|
alpnSupportedXFTPhandshakes :: [ALPN]
|
|
alpnSupportedXFTPhandshakes = [xftpALPNv1]
|
|
|
|
xftpALPNv1 :: ALPN
|
|
xftpALPNv1 = "xftp/1"
|
|
|
|
data XFTPClientHello = XFTPClientHello
|
|
{ -- | a random string sent by the client to the server to prove that server has identity certificate
|
|
webChallenge :: Maybe ByteString
|
|
}
|
|
|
|
data XFTPServerHandshake = XFTPServerHandshake
|
|
{ xftpVersionRange :: VersionRangeXFTP,
|
|
sessionId :: SessionId,
|
|
-- | pub key to agree shared secrets for command authorization and entity ID encryption.
|
|
authPubKey :: CertChainPubKey,
|
|
-- | signed identity challenge from XFTPClientHello
|
|
webIdentityProof :: Maybe C.ASignature
|
|
}
|
|
|
|
data XFTPClientHandshake = XFTPClientHandshake
|
|
{ -- | agreed XFTP server protocol version
|
|
xftpVersion :: VersionXFTP,
|
|
-- | server identity - CA certificate fingerprint
|
|
keyHash :: C.KeyHash
|
|
}
|
|
|
|
instance Encoding XFTPClientHello where
|
|
smpEncode XFTPClientHello {webChallenge} = smpEncode webChallenge
|
|
smpP = do
|
|
webChallenge <- smpP
|
|
forM_ webChallenge $ \challenge -> unless (B.length challenge == 32) $ fail "bad XFTPClientHello webChallenge"
|
|
Tail _compat <- smpP
|
|
pure XFTPClientHello {webChallenge}
|
|
|
|
instance Encoding XFTPClientHandshake where
|
|
smpEncode XFTPClientHandshake {xftpVersion, keyHash} =
|
|
smpEncode (xftpVersion, keyHash)
|
|
smpP = do
|
|
(xftpVersion, keyHash) <- smpP
|
|
Tail _compat <- smpP
|
|
pure XFTPClientHandshake {xftpVersion, keyHash}
|
|
|
|
instance Encoding XFTPServerHandshake where
|
|
smpEncode XFTPServerHandshake {xftpVersionRange, sessionId, authPubKey, webIdentityProof} =
|
|
smpEncode (xftpVersionRange, sessionId, authPubKey, C.signatureBytes webIdentityProof)
|
|
smpP = do
|
|
(xftpVersionRange, sessionId, authPubKey) <- smpP
|
|
webIdentityProof <- optional $ C.decodeSignature <$?> smpP
|
|
Tail _compat <- smpP
|
|
pure XFTPServerHandshake {xftpVersionRange, sessionId, authPubKey, webIdentityProof}
|
|
|
|
sendEncFile :: Handle -> (Builder -> IO ()) -> LC.SbState -> Word32 -> IO ()
|
|
sendEncFile h send = go
|
|
where
|
|
go sbState 0 = do
|
|
let authTag = BA.convert (LC.sbAuth sbState)
|
|
send $ byteString authTag
|
|
go sbState sz =
|
|
getFileChunk h sz >>= \ch -> do
|
|
let (encCh, sbState') = LC.sbEncryptChunk sbState ch
|
|
send (byteString encCh) `E.catch` \(e :: E.SomeException) -> print e >> E.throwIO e
|
|
go sbState' $ sz - fromIntegral (B.length ch)
|
|
|
|
receiveFile :: (Int -> IO ByteString) -> XFTPRcvChunkSpec -> ExceptT XFTPErrorType IO ()
|
|
receiveFile getBody chunk = ExceptT $ runExceptT (receiveFile_ receive chunk) `E.catches` handlers
|
|
where
|
|
receive h sz = hReceiveFile getBody h sz >>= \sz' -> pure $ if sz' == 0 then Right () else Left SIZE
|
|
handlers =
|
|
[ E.Handler $ \(e :: HTTP2Error) -> logWarn (err e) $> Left TIMEOUT,
|
|
E.Handler $ \(e :: E.SomeException) -> logError (err e) $> Left FILE_IO
|
|
]
|
|
err e = "receiveFile error: " <> tshow e
|
|
|
|
receiveEncFile :: (Int -> IO ByteString) -> LC.SbState -> XFTPRcvChunkSpec -> ExceptT XFTPErrorType IO ()
|
|
receiveEncFile getBody = receiveFile_ . receive
|
|
where
|
|
receive sbState h sz = first err <$> receiveSbFile getBody h sbState sz
|
|
err RFESize = SIZE
|
|
err RFECrypto = CRYPTO
|
|
|
|
data ReceiveFileError = RFESize | RFECrypto
|
|
|
|
receiveSbFile :: (Int -> IO ByteString) -> Handle -> LC.SbState -> Word32 -> IO (Either ReceiveFileError ())
|
|
receiveSbFile getBody h = receive
|
|
where
|
|
receive sbState sz = do
|
|
ch <- getBody fileBlockSize
|
|
let chSize = fromIntegral $ B.length ch
|
|
if
|
|
| chSize > sz + authSz -> pure $ Left RFESize
|
|
| chSize > 0 -> do
|
|
let (ch', rest) = B.splitAt (fromIntegral sz) ch
|
|
(decCh, sbState') = LC.sbDecryptChunk sbState ch'
|
|
sz' = sz - fromIntegral (B.length ch')
|
|
B.hPut h decCh
|
|
if sz' > 0
|
|
then receive sbState' sz'
|
|
else do
|
|
let tag' = B.take C.authTagSize rest
|
|
tagSz = B.length tag'
|
|
tag = LC.sbAuth sbState'
|
|
tag'' <- if tagSz == C.authTagSize then pure tag' else (tag' <>) <$> getBody (C.authTagSize - tagSz)
|
|
pure $ if BA.constEq tag'' tag then Right () else Left RFECrypto
|
|
| otherwise -> pure $ Left RFESize
|
|
authSz = fromIntegral C.authTagSize
|
|
|
|
receiveFile_ :: (Handle -> Word32 -> IO (Either XFTPErrorType ())) -> XFTPRcvChunkSpec -> ExceptT XFTPErrorType IO ()
|
|
receiveFile_ receive XFTPRcvChunkSpec {filePath, chunkSize, chunkDigest} = do
|
|
ExceptT $ withFile filePath WriteMode (`receive` chunkSize)
|
|
digest' <- liftIO $ LC.sha256Hash <$> LB.readFile filePath
|
|
when (digest' /= chunkDigest) $ throwE DIGEST
|
|
|
|
data XFTPErrorType
|
|
= -- | incorrect block format, encoding or signature size
|
|
BLOCK
|
|
| -- | incorrect SMP session ID (TLS Finished message / tls-unique binding RFC5929)
|
|
SESSION
|
|
| -- | incorrect handshake command
|
|
HANDSHAKE
|
|
| -- | SMP command is unknown or has invalid syntax
|
|
CMD {cmdErr :: CommandError}
|
|
| -- | command authorization error - bad signature or non-existing SMP queue
|
|
AUTH
|
|
| -- | command with the entity that was blocked
|
|
BLOCKED {blockInfo :: BlockingInfo}
|
|
| -- | incorrent file size
|
|
SIZE
|
|
| -- | storage quota exceeded
|
|
QUOTA
|
|
| -- | incorrent file digest
|
|
DIGEST
|
|
| -- | file encryption/decryption failed
|
|
CRYPTO
|
|
| -- | no expected file body in request/response or no file on the server
|
|
NO_FILE
|
|
| -- | unexpected file body
|
|
HAS_FILE
|
|
| -- | file IO error
|
|
FILE_IO
|
|
| -- | file sending or receiving timeout
|
|
TIMEOUT
|
|
| -- | internal server error
|
|
INTERNAL
|
|
| -- | used internally, never returned by the server (to be removed)
|
|
DUPLICATE_ -- not part of SMP protocol, used internally
|
|
deriving (Eq, Show)
|
|
|
|
instance StrEncoding XFTPErrorType where
|
|
strEncode = \case
|
|
BLOCK -> "BLOCK"
|
|
SESSION -> "SESSION"
|
|
HANDSHAKE -> "HANDSHAKE"
|
|
CMD e -> "CMD " <> bshow e
|
|
AUTH -> "AUTH"
|
|
BLOCKED info -> "BLOCKED " <> strEncode info
|
|
SIZE -> "SIZE"
|
|
QUOTA -> "QUOTA"
|
|
DIGEST -> "DIGEST"
|
|
CRYPTO -> "CRYPTO"
|
|
NO_FILE -> "NO_FILE"
|
|
HAS_FILE -> "HAS_FILE"
|
|
FILE_IO -> "FILE_IO"
|
|
TIMEOUT -> "TIMEOUT"
|
|
INTERNAL -> "INTERNAL"
|
|
DUPLICATE_ -> "DUPLICATE_"
|
|
|
|
strP =
|
|
A.takeTill (== ' ') >>= \case
|
|
"BLOCK" -> pure BLOCK
|
|
"SESSION" -> pure SESSION
|
|
"HANDSHAKE" -> pure HANDSHAKE
|
|
"CMD" -> CMD <$> parseRead1
|
|
"AUTH" -> pure AUTH
|
|
"BLOCKED" -> BLOCKED <$> _strP
|
|
"SIZE" -> pure SIZE
|
|
"QUOTA" -> pure QUOTA
|
|
"DIGEST" -> pure DIGEST
|
|
"CRYPTO" -> pure CRYPTO
|
|
"NO_FILE" -> pure NO_FILE
|
|
"HAS_FILE" -> pure HAS_FILE
|
|
"FILE_IO" -> pure FILE_IO
|
|
"TIMEOUT" -> pure TIMEOUT
|
|
"INTERNAL" -> pure INTERNAL
|
|
"DUPLICATE_" -> pure DUPLICATE_
|
|
_ -> fail "bad error type"
|
|
|
|
instance Encoding XFTPErrorType where
|
|
smpEncode = \case
|
|
BLOCK -> "BLOCK"
|
|
SESSION -> "SESSION"
|
|
HANDSHAKE -> "HANDSHAKE"
|
|
CMD err -> "CMD " <> smpEncode err
|
|
AUTH -> "AUTH"
|
|
BLOCKED info -> "BLOCKED " <> smpEncode info
|
|
SIZE -> "SIZE"
|
|
QUOTA -> "QUOTA"
|
|
DIGEST -> "DIGEST"
|
|
CRYPTO -> "CRYPTO"
|
|
NO_FILE -> "NO_FILE"
|
|
HAS_FILE -> "HAS_FILE"
|
|
FILE_IO -> "FILE_IO"
|
|
TIMEOUT -> "TIMEOUT"
|
|
INTERNAL -> "INTERNAL"
|
|
DUPLICATE_ -> "DUPLICATE_"
|
|
|
|
smpP =
|
|
A.takeTill (== ' ') >>= \case
|
|
"BLOCK" -> pure BLOCK
|
|
"SESSION" -> pure SESSION
|
|
"HANDSHAKE" -> pure HANDSHAKE
|
|
"CMD" -> CMD <$> _smpP
|
|
"AUTH" -> pure AUTH
|
|
"BLOCKED" -> BLOCKED <$> _smpP
|
|
"SIZE" -> pure SIZE
|
|
"QUOTA" -> pure QUOTA
|
|
"DIGEST" -> pure DIGEST
|
|
"CRYPTO" -> pure CRYPTO
|
|
"NO_FILE" -> pure NO_FILE
|
|
"HAS_FILE" -> pure HAS_FILE
|
|
"FILE_IO" -> pure FILE_IO
|
|
"TIMEOUT" -> pure TIMEOUT
|
|
"INTERNAL" -> pure INTERNAL
|
|
"DUPLICATE_" -> pure DUPLICATE_
|
|
_ -> fail "bad error type"
|
|
|
|
$(J.deriveJSON (sumTypeJSON id) ''XFTPErrorType)
|