From 7ca1f4e508b7834a1fa86d3f25067e2928319e9c Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sat, 7 Nov 2026 15:23:45 +0000 Subject: [PATCH] agent: complete file uploads when the app leaves the foreground (#2010) * agent: continue XFTP uploads started in foreground while suspending * tests: verify XFTP upload stage of agent suspension * docs: plan for XFTP uploads while suspending * docs: revise plan for XFTP uploads while suspending * docs: revise plan wording for XFTP uploads while suspending * agent: exclude ended and stopped XFTP uploads from suspension wait * tests: verify suspension wait for ended and stopped XFTP uploads * docs: plan decisions for ended XFTP uploads while suspending * agent: simplify tryBeginAgentOperation with orElse * agent: rename upload wait functions by the conditions they check * tests: verify database check and deleted rows at suspension start * docs: plan residual case, conditions and names for XFTP suspension * docs: plan tests for the remaining XFTP suspension paths * tests: verify the remaining XFTP suspension paths * docs: update plan line references and the query error test * tests: foreground agent before checking the query error result * docs: plan line references after merging master and stage diagrams * agent: one check of the begun database operation when removing deleted or failed counted files * agent: suspendAgent returns before the database query of counted files * tests: closed database store tests excluded from PostgreSQL agent builds * tests: suspension timeout test waits for the timeout and bounds SUSPENDED * tests: one setup helper for suspension tests with a small sent file, one for the bounded SUSPENDED wait * tests: suspendAgent returns while the agent database is in use * docs: plan states the forked removal at suspension start, uncounted redirects, offline files and the changed tests * docs: plan links the simplex-chat pull request * agent: suspend timeout starts when suspendAgent returns, not after the database query of counted files * docs: plan states the separate removal thread and the TTL cleanup interval --- ...026-10-23-xftp-uploads-while-suspending.md | 206 +++++++++++++++ src/Simplex/FileTransfer/Agent.hs | 22 +- src/Simplex/Messaging/Agent.hs | 19 +- src/Simplex/Messaging/Agent/Client.hs | 68 ++++- .../Messaging/Agent/Store/AgentStore.hs | 5 + tests/AgentTests/FunctionalAPITests.hs | 37 ++- tests/XFTPAgent.hs | 237 +++++++++++++++++- 7 files changed, 569 insertions(+), 25 deletions(-) create mode 100644 plans/2026-10-23-xftp-uploads-while-suspending.md diff --git a/plans/2026-10-23-xftp-uploads-while-suspending.md b/plans/2026-10-23-xftp-uploads-while-suspending.md new file mode 100644 index 000000000..888746969 --- /dev/null +++ b/plans/2026-10-23-xftp-uploads-while-suspending.md @@ -0,0 +1,206 @@ +# XFTP uploads while the agent is suspending + +On iOS, a file sent shortly before the app left the foreground was delivered only after the app was opened again. Once the agent began to suspend, the next chunk was not started, and the result of the chunk in flight was recorded only in foreground. +While the agent is suspending, it continues the uploads that it started in foreground. Their files are counted by a new operation, `AOSndFiles`, with its own count and set, in a stage between `AORcvNetwork` and `AOMsgDelivery`. The next stage is entered after these files are deleted by the chat, removed by TTL cleanup or `closeAgentClient`, or removed at the start of the suspension when their rows are absent or flagged deleted or failed. At the suspend timeout, the stage of `AOSndNetwork` is entered if the agent is still suspending. + +Chat side: https://github.com/simplex-chat/simplex-chat/pull/7966 + +## Decisions + +- Stages of `suspendAgent` with a timeout above 0: + + | Order | Operation | Count | At count 0 | + |---|---|---|---| + | 1 | `AORcvNetwork` | received transmission batches in processing by the subscriber | `AOSndFiles` suspended | + | 2 | `AOSndFiles` | counted files | `AOMsgDelivery` suspended | + | 3 | `AOMsgDelivery` | submitted messages not yet taken by delivery workers | `AOSndNetwork` suspended | + | 4 | `AOSndNetwork` | delivery workers and async command workers processing work | `AODatabase` suspended | + | 5 | `AODatabase` | database transactions | `SUSPENDED` sent; agent state `ASSuspended` | + + - The actions of "At count 0" are taken only while the agent state is `ASSuspending`. + - `AONtfNetwork` is suspended at the start; the stages are entered independently of its count. + - After the timeout, if the agent is still suspending, stage 4 is entered whatever stage is in progress; stage 5 is entered when the count of stage 4 is 0, as in the table. + + ```mermaid + stateDiagram-v2 + [*] --> RcvNetwork: suspendAgent, timeout above 0 + RcvNetwork --> SndFiles: AORcvNetwork count 0 + SndFiles --> MsgDelivery: AOSndFiles count 0 + MsgDelivery --> SndNetwork: AOMsgDelivery count 0 + SndNetwork --> Database: AOSndNetwork count 0 + Database --> Suspended: AODatabase count 0, SUSPENDED sent + RcvNetwork --> SndNetwork: timeout + SndFiles --> SndNetwork: timeout + MsgDelivery --> SndNetwork: timeout + Suspended --> [*] + ``` + +- Counted files, the set `sndFilesInProgress`: + - inserted: files started by `xftpStartSendFile'` while the agent state is `ASForeground`; + - removed: + - files deleted by `deleteSndFilesInternal`, after its database work, also when an exception is thrown; + - files expired by TTL cleanup, before their prefix path and row are deleted; + - at the start of a suspension with a timeout above 0, in a thread forked by `suspendAgent`, while `AODatabase` is not suspended: files whose rows are absent or flagged `deleted = 1` or `failed = 1`, found by one query; the suspended flag of `AODatabase` is checked and the operation is begun in one STM transaction; `suspendAgent` returns, and the suspend timeout starts, before this query; + - every file, by `closeAgentClient`. + - kept until the chat deletes them: files with complete or error rows; the file descriptions are sent while these files are counted. + - kept until TTL cleanup, `closeAgentClient` or process exit: counted files completed, or set to an error status, by another process; each suspension in between waits for the suspend timeout. + - kept until they are uploaded or removed: counted files that cannot be uploaded, for example offline; TTL cleanup removes such a file at its first pass after `sndFilesTTL` (1 day) from creation, at most `cleanupInterval` (5 min) later; each suspension in between waits in the files stage until the suspend timeout. +- `deleteSndFilesInternal` is run by `xftpDeleteSndFileInternal`, which is called by the chat after `SFDONE` and `SFERR`. +- Uncounted files: + - files resumed at worker start; + - files started while suspending or suspended; + - among them, the redirect upload of a standalone file, started by the chat on `SFDONE` of its source file (chat `Subscriber.hs:269`, `Internal.hs:497`): its upload stops when the source file is deleted, unless other counted files remain. +- Condition of the send prepare and upload workers, `uploadsAllowed`: + + | Agent state | Counted files | `AOSndNetwork` suspended | Condition | + |---|---|---|---| + | `ASForeground` | any | any | true | + | `ASSuspending` | above 0 | no | true | + | `ASSuspending` | 0 | any | false | + | `ASSuspending` | any | yes | false | + | `ASSuspended` | any | any | false | + + - While the condition is false, the send prepare and upload workers wait until the agent state is `ASForeground`. + - While the condition is true, uncounted files are uploaded too. +- Receive and delete workers: unchanged. +- `foregroundAgent`: suspended flags cleared; counted files kept. + +``` +xftpStartSendFile' f ASForeground -> add f +deleteSndFilesInternal fs rows deleted or marked deleted; finally: remove each f +deleteSndFilesExpired f remove f; prefix path and row deleted +suspendAgent, timeout > 0 forked thread: AODatabase not suspended -> begin AODatabase; remove each f whose row is absent, deleted or failed; end AODatabase +closeAgentClient remove each f +add f f not in set -> insert f; count + 1 +remove f f in set -> delete f; endAgentOperation AOSndFiles +``` + +```mermaid +flowchart LR + start["xftpStartSendFile' in ASForeground"] --> add["add f"] + delete["deleteSndFilesInternal, finally"] --> remove["remove f"] + ttl["TTL cleanup"] --> remove + suspend["suspension start, forked thread: row absent, deleted or failed"] --> remove + close["closeAgentClient"] --> remove + remove --> stage["endAgentOperation AOSndFiles"] +``` + +## Mechanics + +- `src/Simplex/Messaging/Agent/Client.hs`: + - `AgentClient` (:379-380), fields created in `newAgentClient` (:550-551): + - `sndFilesOp :: TVar AgentOpState`; + - `sndFilesInProgress :: TVar (Set SndFileId)`. + - `AgentOperation`: `AOSndFiles` (:476). + - `sndFilesOp` added to: + - `agentOpSel` (:483); + - `agentOperations` (:489). + - `closeAgentClient` (:1024): every counted file removed by `removeSndFileInProgress`. + - `endAgentOperation` (:2363-2377): at the end of `AORcvNetwork`, `AOSndFiles` is suspended; at the end of `AOSndFiles`, `suspendMsgDelivery` is called. + - `suspendMsgDelivery` (:2379-2382): `AOMsgDelivery` is suspended; `suspendSendingAndDatabase` is called when its count is 0. + - new: + - `addSndFileInProgress` (:2414-2417); + - `removeSndFileInProgress` (:2419-2422); + - `removeDeletedOrFailedSndFilesInProgress` (:2424-2435): when the set is not empty and `AODatabase` is begun by `tryBeginAgentOperation`, the counted files missing from `getNotDeletedOrFailedSndFileIds` are removed; on a query error, the error is logged and the set is unchanged; under `E.mask`, the begun operation is ended by `E.finally`; + - `tryBeginAgentOperation` (:2446-2447): `beginAgentOperation`, or `False` by `orElse` while the operation is suspended; `True` when begun; + - `waitUntilUploadsAllowed` (:2472-2482), with the condition `uploadsAllowed`. +- `src/Simplex/Messaging/Agent/Store/AgentStore.hs`, new: `getNotDeletedOrFailedSndFileIds` (:3898-3900), the entity IDs of `snd_files` rows with `deleted = 0` and `failed = 0`. +- `src/Simplex/Messaging/Agent.hs`: + - `suspendAgent`: + - stages (:3446-3448): `AORcvNetwork`, `AOSndFiles`, then `suspendMsgDelivery`; + - `removeDeletedOrFailedSndFilesInProgress` (:3451), in a thread forked while the agent is suspending, separate from the timeout thread (:3452-3457); a removal that ends after the timeout holds the `AODatabase` count until it ends, and `SUSPENDED` is sent then. + - `deleteSndFilesExpired` (:3548), local to `cleanupManager`: `removeSndFileInProgress`. +- `src/Simplex/FileTransfer/Agent.hs`: + - `xftpStartSendFile'` (:407): `addSndFileInProgress` when `ASForeground`. + - `assertUploadsAllowed` in place of `assertAgentForeground`: + - `runXFTPSndPrepareWorker` (:424, :487, :507); + - `runXFTPSndWorker` (:528, :549, :560). + - `uploadFileChunk` (:562): `waitUntilUploadsAllowed` in place of `waitUntilForeground`. + - `deleteSndFilesInternal` (:663): `removeSndFileInProgress` in `E.finally`. + - `assertUploadsAllowed` (:749-752): `throwWhenInactive`, then `waitUntilUploadsAllowed`. + +## Tests + +`tests/XFTPAgent.hs`, "agent XFTP API": +- Helpers: + - `withSndFile` (:822): file of 128 KB; agent 1 with the test's configuration; workers started; the file sent to 1 recipient; + - `shouldSuspend` (:831): `SUSPENDED` within 5 s; events of the file before it are skipped. +- "should complete sending file while agent is suspending" (:644): + - suspend timeout 60 s; + - all `SFPROG` within 30 s, then `SFDONE`; + - events absent for 300 ms; + - after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s. +- "should suspend on timeout when file is not sent" (:659): + - XFTP server down, suspend timeout 0.5 s: events absent for 300 ms, then `SUSPENDED` within 5 s; + - server started: events absent for 1 s; + - after `foregroundAgent`, `SFDONE`. +- "should not wait for file started while agent is suspending" (:676): + - file 1 started, suspend timeout 60 s, file 2 started; + - `SFDONE` of file 1 within 30 s; + - after `xftpDeleteSndFileInternal` of file 1, `shouldSuspend` for file 2. +- "should continue sending file when another file is deleted twice while agent is suspending" (:690): + - file 1 of 128 KB and file 2 of 17 MB started, suspend timeout 60 s; + - `xftpDeleteSndFileInternal` of file 1, twice; + - `SFDONE` of file 2 within 30 s; `SUSPENDED` before it: test failure; + - after `xftpDeleteSndFileInternal` of file 2, `shouldSuspend` for file 1. +- "should wait for complete file until it is deleted while agent is suspending" (:706): + - `withSndFile`; `SFDONE` within 30 s; + - suspend timeout 60 s; events absent for 300 ms; + - after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s. +- `testXFTPAgentSuspendAfter` (:715), for the next four tests: + - XFTP server down; `withSndFile`; + - the test's step; + - suspend timeout 60 s; `shouldSuspend`. +- "should not wait for file deleted from database while agent is suspending" (:102): `DELETE FROM snd_files` through the agent's store. +- "should not wait for file marked deleted while agent is suspending" (:104): `UPDATE snd_files SET deleted = 1` through the agent's store. +- "should not wait for file marked failed while agent is suspending" (:106): `UPDATE snd_files SET failed = 1` through the agent's store. +- "should not wait for file after agent client is closed" (:108): `disconnectAgentClient`, then `resumeAgentClient`. +- "should not access database when suspending while database is suspended" (:722): + - XFTP server down; `withSndFile`; + - suspend timeout 0; the file row deleted by the test; + - suspend timeout 60 s: after 1 s, the set still contains the file. +- "should not wait for database when suspending" (:731): + - XFTP server down; `withSndFile`; + - while the test holds the agent store connection: `suspendAgent` with timeout 0.5 s returns within 1 s; `AOSndNetwork` is suspended by the timeout within 5 s; + - after the test releases the connection, `shouldSuspend`. +- "should not wait for file removed by TTL cleanup while agent is suspending" (:739): + - cleanup after 1 s and every 1 s; `sndFilesTTL` 1 s; + - XFTP server down; `withSndFile`; + - suspend timeout 60 s: `SUSPENDED` within 10 s, after the cleanup removes the expired file; events of the file before it are skipped. +- "should not wait for file resumed at worker start" (:746): + - XFTP server down; file of 128 KB started by agent 1; + - agent 2 on the same database resumes it at `xftpStartWorkers`; + - after 0.5 s, suspend timeout 60 s; `shouldSuspend`. +- "should stop uploads when sending network is suspended while agent is suspending" (:758): + - `withSndFile`; `AOSndNetwork` begun by the test; + - suspend timeout 1 µs: `SFDONE` absent for 3 s; + - after the test ends `AOSndNetwork`, `shouldSuspend`. +- "should retry creating file chunks while agent is suspending" (:767): + - XFTP server down; `withSndFile`; suspend timeout 60 s; + - server started after 3 s: `SFDONE` within 30 s; + - after `xftpDeleteSndFileInternal`, `shouldSuspend`. +- "should retry uploading file chunks while agent is suspending" (:777): + - file of 17 MB started; the server is stopped after the first `SFPROG`; + - suspend timeout 60 s; server started after 1 s: `SFDONE` within 30 s; + - after `xftpDeleteSndFileInternal`, `shouldSuspend`. +- "should remove counted file when its deletion fails" (:794): + - XFTP server down; `withSndFile`; + - agent store closed: `xftpDeleteSndFileInternal` throws, and the set is empty. +- "should keep counted files when database query fails at suspension start" (:801): + - XFTP server down; `withSndFile`; + - agent store closed; suspend timeout 60 s; + - after 1 s, the set still contains the file; `foregroundAgent` before the comparison. +- The two tests above are excluded from builds with `client_postgres` (`#if !defined(dbPostgres)`, :117-120): a closed PostgreSQL store has no pool connections, and its `withTransaction` waits for one. +- "should wait for counted file again after agent is in foreground" (:811): + - XFTP server down; `withSndFile`; + - suspend timeout 0.5 s; `shouldSuspend`; + - `foregroundAgent`; suspend timeout 60 s: `SUSPENDED` absent for 1 s; + - after `xftpDeleteSndFileInternal`, `shouldSuspend`. + +`tests/AgentTests/FunctionalAPITests.hs`, "Suspending agent": +- "should send messages queued after file is sent before agent is suspended" (:3177): + - b starts a file, suspend timeout 60 s; + - once `sndFilesOp` is suspended, the `opSuspended` flags of `rcvNetworkOp`, `sndFilesOp`, `msgDeliveryOp`: `[True, True, False]`; + - after `SFDONE`, `sendMessage` and `xftpDeleteSndFileInternal`: `[True, True, True]`; + - `SENT`, then `SUSPENDED` within 5 s; + - a receives the message. diff --git a/src/Simplex/FileTransfer/Agent.hs b/src/Simplex/FileTransfer/Agent.hs index 0f7b19b65..f3976c47a 100644 --- a/src/Simplex/FileTransfer/Agent.hs +++ b/src/Simplex/FileTransfer/Agent.hs @@ -404,6 +404,7 @@ xftpPrepareSendDescription' c userId (ValidFileDescription fdDirect@FileDescript xftpStartSendFile' :: AgentClient -> SndFileId -> AM () xftpStartSendFile' c sndFileEntityId = do withStore c (`startPreparedSndFile` sndFileEntityId) + atomically . whenM ((ASForeground ==) <$> readTVar (agentState c)) $ addSndFileInProgress c sndFileEntityId lift . void $ getXFTPSndWorker True c Nothing resumeXFTPSndWork :: AgentClient -> Maybe XFTPServer -> AM' () @@ -420,7 +421,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do cfg <- asks config forever $ do lift $ waitForWork doWork - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c runXFTPOperation cfg where runXFTPOperation :: AgentConfig -> AM () @@ -483,7 +484,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do SndFileChunkReplica {server} : _ -> Right server createChunk :: Int -> Maybe Word32 -> SndFileChunk -> AM (ProtocolServer 'PXFTP) createChunk numRecipients' storageHours ch = do - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c (replica, ProtoServerWithAuth srv _) <- tryCreate withStore' c $ \db -> createSndFileReplica db ch replica pure srv @@ -503,7 +504,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do retryLoop loop triedAllSrvs e = do flip catchAllErrors (\_ -> pure ()) $ do when (triedAllSrvs && serverHostError e) $ notify c sndFileEntityId $ SFWARN e - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c loop createWithNextSrv triedHosts = do deleted <- withStore' c $ \db -> getSndFileDeleted db sndFileId @@ -524,7 +525,7 @@ runXFTPSndWorker c srv Worker {doWork} = do cfg <- asks config forever $ do lift $ waitForWork doWork - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c runXFTPOperation cfg where runXFTPOperation :: AgentConfig -> AM () @@ -545,7 +546,7 @@ runXFTPSndWorker c srv Worker {doWork} = do when (serverHostError e) $ notify c sndFileEntityId $ SFWARN e liftIO $ closeXFTPServerClient c userId server digest withStore' c $ \db -> updateSndChunkReplicaDelay db sndChunkReplicaId replicaDelay - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c loop retryDone e = do incXFTPServerStat c userId srv uploadErrs @@ -556,9 +557,9 @@ runXFTPSndWorker c srv Worker {doWork} = do fsFilePath <- lift $ toFSFilePath filePath unlessM (doesFileExist fsFilePath) $ throwE $ FILE NO_FILE let chunkSpec' = chunkSpec {filePath = fsFilePath} :: XFTPChunkSpec - liftIO $ assertAgentForeground c + liftIO $ assertUploadsAllowed c agentXFTPUploadChunk c userId chunkDigest replica' chunkSpec' - liftIO $ waitUntilForeground c + liftIO $ waitUntilUploadsAllowed c sf@SndFile {sndFileEntityId, prefixPath, chunks} <- withStore c $ \db -> do lockSndFileForUpdate db sndFileId updateSndChunkReplicaStatus db sndChunkReplicaId SFRSUploaded @@ -659,7 +660,7 @@ deleteSndFileInternal :: AgentClient -> SndFileId -> AM' () deleteSndFileInternal c sndFileEntityId = deleteSndFilesInternal c [sndFileEntityId] deleteSndFilesInternal :: AgentClient -> [SndFileId] -> AM' () -deleteSndFilesInternal c sndFileEntityIds = do +deleteSndFilesInternal c sndFileEntityIds = flip E.finally (atomically $ mapM_ (removeSndFileInProgress c) sndFileEntityIds) $ do sndFiles <- rights <$> withStoreBatch c (\db -> map (fmap (first storeError) . getSndFileByEntityId db) sndFileEntityIds) let (toDelete, toMarkDeleted) = partition fileComplete sndFiles workPath <- getXFTPWorkPath @@ -744,3 +745,8 @@ assertAgentForeground :: AgentClient -> IO () assertAgentForeground c = do throwWhenInactive c waitUntilForeground c + +assertUploadsAllowed :: AgentClient -> IO () +assertUploadsAllowed c = do + throwWhenInactive c + waitUntilUploadsAllowed c diff --git a/src/Simplex/Messaging/Agent.hs b/src/Simplex/Messaging/Agent.hs index a474c86a4..0e3a1bce9 100644 --- a/src/Simplex/Messaging/Agent.hs +++ b/src/Simplex/Messaging/Agent.hs @@ -3444,15 +3444,17 @@ suspendAgent c@AgentClient {agentState = as} maxDelay = do writeTVar as ASSuspending suspendOperation c AONtfNetwork $ pure () suspendOperation c AORcvNetwork $ - suspendOperation c AOMsgDelivery $ - suspendSendingAndDatabase c + suspendOperation c AOSndFiles $ + suspendMsgDelivery c readTVar as - when (state == ASSuspending) . void . forkIO $ do - threadDelay maxDelay - -- liftIO $ putStrLn "suspendAgent after timeout" - atomically . whenSuspending c $ do - -- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase" - suspendSendingAndDatabase c + when (state == ASSuspending) $ do + void . forkIO $ removeDeletedOrFailedSndFilesInProgress c + void . forkIO $ do + threadDelay maxDelay + -- liftIO $ putStrLn "suspendAgent after timeout" + atomically . whenSuspending c $ do + -- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase" + suspendSendingAndDatabase c execAgentStoreSQL :: AgentClient -> Text -> AE [Text] execAgentStoreSQL c sql = withAgentEnv c $ withStore' c (`execSQL` sql) @@ -3544,6 +3546,7 @@ cleanupManager c@AgentClient {subQ} = do sndFilesTTL <- asks $ sndFilesTTL . config sndExpired <- withStore' c (`getSndFilesExpired` sndFilesTTL) forM_ sndExpired $ \(dbId, entId, p) -> flip catchAllErrors (notify entId . SFERR) $ do + atomically $ removeSndFileInProgress c entId lift . forM_ p $ removePath <=< toFSFilePath withStore' c (`deleteSndFile'` dbId) deleteSndFilesDeleted = do diff --git a/src/Simplex/Messaging/Agent/Client.hs b/src/Simplex/Messaging/Agent/Client.hs index f79de284c..22ae263b1 100644 --- a/src/Simplex/Messaging/Agent/Client.hs +++ b/src/Simplex/Messaging/Agent/Client.hs @@ -156,9 +156,14 @@ module Simplex.Messaging.Agent.Client waitUntilForeground, waitWhileSuspended, suspendSendingAndDatabase, + suspendMsgDelivery, suspendOperation, notifySuspended, whenSuspending, + addSndFileInProgress, + removeSndFileInProgress, + removeDeletedOrFailedSndFilesInProgress, + waitUntilUploadsAllowed, withStore, withStore', withStoreBatch, @@ -235,7 +240,7 @@ import Simplex.FileTransfer.Description (ChunkReplicaId (..), FileDigest (..), k import Simplex.FileTransfer.Protocol (FileInfo (..), FileResponse) import Simplex.FileTransfer.Transport (XFTPErrorType (DIGEST), XFTPRcvChunkSpec (..), XFTPVersion) import qualified Simplex.FileTransfer.Transport as XFTP -import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..)) +import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..), SndFileId) import Simplex.FileTransfer.Util (uniqueCombine) import Simplex.Messaging.Agent.Env.SQLite import Simplex.Messaging.Agent.Lock @@ -371,6 +376,8 @@ data AgentClient = AgentClient asyncCmdWorkers :: TMap (ConnId, Maybe SMPServer) Worker, ntfNetworkOp :: TVar AgentOpState, rcvNetworkOp :: TVar AgentOpState, + sndFilesOp :: TVar AgentOpState, + sndFilesInProgress :: TVar (Set SndFileId), msgDeliveryOp :: TVar AgentOpState, sndNetworkOp :: TVar AgentOpState, databaseOp :: TVar AgentOpState, @@ -466,19 +473,20 @@ runWorkerAsync Worker {action} work = where start = atomically . putTMVar action . Just =<< mkWeakThreadId =<< forkIO work -data AgentOperation = AONtfNetwork | AORcvNetwork | AOMsgDelivery | AOSndNetwork | AODatabase +data AgentOperation = AONtfNetwork | AORcvNetwork | AOSndFiles | AOMsgDelivery | AOSndNetwork | AODatabase deriving (Eq, Show) agentOpSel :: AgentOperation -> (AgentClient -> TVar AgentOpState) agentOpSel = \case AONtfNetwork -> ntfNetworkOp AORcvNetwork -> rcvNetworkOp + AOSndFiles -> sndFilesOp AOMsgDelivery -> msgDeliveryOp AOSndNetwork -> sndNetworkOp AODatabase -> databaseOp agentOperations :: [AgentClient -> TVar AgentOpState] -agentOperations = [ntfNetworkOp, rcvNetworkOp, msgDeliveryOp, sndNetworkOp, databaseOp] +agentOperations = [ntfNetworkOp, rcvNetworkOp, sndFilesOp, msgDeliveryOp, sndNetworkOp, databaseOp] data AgentOpState = AgentOpState {opSuspended :: !Bool, opsInProgress :: !Int} @@ -539,6 +547,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf asyncCmdWorkers <- TM.emptyIO ntfNetworkOp <- newTVarIO $ AgentOpState False 0 rcvNetworkOp <- newTVarIO $ AgentOpState False 0 + sndFilesOp <- newTVarIO $ AgentOpState False 0 + sndFilesInProgress <- newTVarIO S.empty msgDeliveryOp <- newTVarIO $ AgentOpState False 0 sndNetworkOp <- newTVarIO $ AgentOpState False 0 databaseOp <- newTVarIO $ AgentOpState False 0 @@ -583,6 +593,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf asyncCmdWorkers, ntfNetworkOp, rcvNetworkOp, + sndFilesOp, + sndFilesInProgress, msgDeliveryOp, sndNetworkOp, databaseOp, @@ -1009,6 +1021,7 @@ closeAgentClient c = do atomically $ SS.clear $ currentSubs c clear subscrConns clear getMsgLocks + atomically $ readTVar (sndFilesInProgress c) >>= mapM_ (removeSndFileInProgress c) where clearWorkers :: Ord k => (AgentClient -> TMap k a) -> IO (Map k a) clearWorkers workers = atomically $ swapTVar (workers c) mempty @@ -2352,8 +2365,10 @@ endAgentOperation :: AgentClient -> AgentOperation -> STM () endAgentOperation c op = endOperation c op $ case op of AONtfNetwork -> pure () AORcvNetwork -> - suspendOperation c AOMsgDelivery $ - suspendSendingAndDatabase c + suspendOperation c AOSndFiles $ + suspendMsgDelivery c + AOSndFiles -> + suspendMsgDelivery c AOMsgDelivery -> suspendSendingAndDatabase c AOSndNetwork -> @@ -2362,6 +2377,11 @@ endAgentOperation c op = endOperation c op $ case op of AODatabase -> notifySuspended c +suspendMsgDelivery :: AgentClient -> STM () +suspendMsgDelivery c = + suspendOperation c AOMsgDelivery $ + suspendSendingAndDatabase c + suspendSendingAndDatabase :: AgentClient -> STM () suspendSendingAndDatabase c = suspendOperation c AOSndNetwork $ @@ -2392,6 +2412,29 @@ whenSuspending :: AgentClient -> STM () -> STM () whenSuspending c = whenM ((== ASSuspending) <$> readTVar (agentState c)) {-# INLINE whenSuspending #-} +addSndFileInProgress :: AgentClient -> SndFileId -> STM () +addSndFileInProgress c fId = do + added <- stateTVar (sndFilesInProgress c) $ \fs -> (S.notMember fId fs, S.insert fId fs) + when added $ modifyTVar' (sndFilesOp c) $ \s -> s {opsInProgress = opsInProgress s + 1} + +removeSndFileInProgress :: AgentClient -> SndFileId -> STM () +removeSndFileInProgress c fId = do + removed <- stateTVar (sndFilesInProgress c) $ \fs -> (S.member fId fs, S.delete fId fs) + when removed $ endAgentOperation c AOSndFiles + +removeDeletedOrFailedSndFilesInProgress :: AgentClient -> IO () +removeDeletedOrFailedSndFilesInProgress c = do + fIds <- readTVarIO $ sndFilesInProgress c + unless (S.null fIds) $ + E.mask $ \restore -> + whenM (atomically $ tryBeginAgentOperation c AODatabase) $ + restore (removeDeletedOrFailed fIds) `E.finally` atomically (endAgentOperation c AODatabase) + where + removeDeletedOrFailed fIds = + E.tryAny (withTransaction (agentClientStore c) getNotDeletedOrFailedSndFileIds) >>= \case + Right fIds' -> atomically $ mapM_ (removeSndFileInProgress c) $ fIds `S.difference` S.fromList fIds' + Left e -> logError $ "removeDeletedOrFailedSndFilesInProgress error: " <> tshow e + beginAgentOperation :: AgentClient -> AgentOperation -> STM () beginAgentOperation c op = do let opVar = agentOpSel op c @@ -2401,6 +2444,9 @@ beginAgentOperation c op = do -- unsafeIOToSTM $ putStrLn $ "beginOperation! " <> show op <> " " <> show (opsInProgress s + 1) writeTVar opVar $! s {opsInProgress = opsInProgress s + 1} +tryBeginAgentOperation :: AgentClient -> AgentOperation -> STM Bool +tryBeginAgentOperation c op = (True <$ beginAgentOperation c op) `orElse` pure False + agentOperationBracket :: MonadUnliftIO m => AgentClient -> AgentOperation -> (AgentClient -> IO ()) -> m a -> m a agentOperationBracket c op check action = E.bracket @@ -2424,6 +2470,18 @@ waitWhileSuspended c = suspended :: Monad m => (TVar AgentState -> m AgentState) -> m Bool suspended rd = (ASSuspended ==) <$> rd (agentState c) +waitUntilUploadsAllowed :: AgentClient -> IO () +waitUntilUploadsAllowed c = atomically $ unlessM uploadsAllowed retry + where + uploadsAllowed = + readTVar (agentState c) >>= \case + ASForeground -> pure True + ASSuspending -> do + AgentOpState {opsInProgress} <- readTVar $ sndFilesOp c + AgentOpState {opSuspended} <- readTVar $ sndNetworkOp c + pure $ opsInProgress > 0 && not opSuspended + ASSuspended -> pure False + withStore' :: AgentClient -> (DB.Connection -> IO a) -> AM a withStore' c action = withStore c $ fmap Right . action {-# INLINE withStore' #-} diff --git a/src/Simplex/Messaging/Agent/Store/AgentStore.hs b/src/Simplex/Messaging/Agent/Store/AgentStore.hs index ccdd80d16..9f350aaa0 100644 --- a/src/Simplex/Messaging/Agent/Store/AgentStore.hs +++ b/src/Simplex/Messaging/Agent/Store/AgentStore.hs @@ -262,6 +262,7 @@ module Simplex.Messaging.Agent.Store.AgentStore getCleanupSndFilesPrefixPaths, getCleanupSndFilesDeleted, getSndFilesExpired, + getNotDeletedOrFailedSndFileIds, createDeletedSndChunkReplica, getNextDeletedSndChunkReplica, updateDeletedSndChunkReplicaDelay, @@ -3894,6 +3895,10 @@ getSndFilesExpired db ttl = do |] (Only cutoffTs) +getNotDeletedOrFailedSndFileIds :: DB.Connection -> IO [SndFileId] +getNotDeletedOrFailedSndFileIds db = + map fromOnly <$> DB.query_ db "SELECT snd_file_entity_id FROM snd_files WHERE deleted = 0 AND failed = 0" + createDeletedSndChunkReplica :: DB.Connection -> UserId -> FileChunkReplica -> FileDigest -> IO () createDeletedSndChunkReplica db userId FileChunkReplica {server, replicaId, replicaKey} chunkDigest = do srvId <- createXFTPServer_ db server diff --git a/tests/AgentTests/FunctionalAPITests.hs b/tests/AgentTests/FunctionalAPITests.hs index 7ed0d17b8..85880041d 100644 --- a/tests/AgentTests/FunctionalAPITests.hs +++ b/tests/AgentTests/FunctionalAPITests.hs @@ -43,6 +43,7 @@ module AgentTests.FunctionalAPITests rfGet, sfGet, nGet, + pGet, getInAnyOrder, (##>), (=##>), @@ -90,7 +91,7 @@ import SMPAgentClient import SMPClient import Simplex.Messaging.Agent hiding (acceptContact, createConnection, deleteConnection, deleteConnections, getConnShortLink, joinConnection, sendMessage, setConnShortLink, subscribeConnection, suspendConnection) import qualified Simplex.Messaging.Agent as A -import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork) +import Simplex.Messaging.Agent.Client (AgentOpState (..), ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork) import Simplex.Messaging.Agent.Env.SQLite (AgentConfig (..), Env (..), InitialAgentServers (..), createAgentStore) import Simplex.Messaging.Agent.Protocol hiding (CON, CONF, INFO, REQ, SENT) import qualified Simplex.Messaging.Agent.Protocol as A @@ -102,6 +103,7 @@ import qualified Simplex.Messaging.Agent.Store.DB as DB import Simplex.Messaging.Agent.Store.Shared (MigrationConfig (..), MigrationConfirmation (..), MigrationError (..)) import Simplex.Messaging.Client (pattern NRMInteractive, NetworkConfig (..), ProtocolClientConfig (..), TransportSessionMode (..), defaultClientConfig, defaultSMPClientConfig, getProtocolClient, sendSMPMessage) import qualified Simplex.Messaging.Crypto as C +import Simplex.Messaging.Crypto.File (CryptoFile (..)) import Simplex.Messaging.Crypto.Ratchet (InitialKeys (..), PQEncryption (..), PQSupport (..), pattern IKPQOff, pattern IKPQOn, pattern PQEncOff, pattern PQEncOn, pattern PQSupportOff, pattern PQSupportOn) import qualified Simplex.Messaging.Crypto.Ratchet as CR import Simplex.Messaging.Encoding @@ -122,10 +124,11 @@ import Simplex.Messaging.Version (VersionRange (..)) import qualified Simplex.Messaging.Version as V import Simplex.Messaging.Version.Internal (Version (..)) import System.Directory (copyFile, removeFile, renameFile) +import System.FilePath (()) import Test.Hspec hiding (fit, it) import UnliftIO import Util -import XFTPClient (testXFTPServer) +import XFTPClient (testXFTPServer, withXFTPServerThreadOn) #if defined(dbPostgres) import Fixtures @@ -502,6 +505,8 @@ functionalAPITests ps = do testSuspendingAgentCompleteSending ps it "should suspend agent on timeout, even if pending messages not sent" $ testSuspendingAgentTimeout ps + it "should send messages queued after file is sent before agent is suspended" $ + testSuspendingAgentCompleteSendingFile ps describe "Batching SMP commands" $ do -- disable this and enable the following test to run tests with coverage it "should subscribe to multiple (200) subscriptions with batching" $ @@ -3169,6 +3174,34 @@ testSuspendingAgentTimeout ps = withAgentClients2 $ \a b -> do ("", "", SUSPENDED) <- nGet b pure () +testSuspendingAgentCompleteSendingFile :: HasTestEnv => (ASrvTransport, AStoreType) -> IO () +testSuspendingAgentCompleteSendingFile ps = + withSmpServer ps $ withXFTPServerThreadOn $ \_ -> withAgentClients2 $ \a b -> runRight_ $ do + (aId, bId) <- makeConnection a b + let filePath = senderFiles "testfile" + liftIO $ B.writeFile filePath "file content" + xftpStartWorkers b (Just senderFiles) + sfId <- xftpSendFile b 1 (CryptoFile filePath Nothing) 1 Nothing + liftIO $ do + suspendAgent b 60000000 + timeout 5000000 (waitSuspendedOp b sndFilesOp) `shouldReturn` Just () + suspendedOps b `shouldReturn` [True, True, False] + ("", _, SFPROG _ _) <- sfGet b + ("", sfId', SFDONE {}) <- sfGet b + liftIO $ sfId' `shouldBe` sfId + msgId <- sendMessage b aId SMP.noMsgFlags "file description" + liftIO $ do + xftpDeleteSndFileInternal b sfId + suspendedOps b `shouldReturn` [True, True, True] + get b ##> ("", aId, SENT msgId) + liftIO $ timeout 5000000 (nGet b) `shouldReturn` Just ("", "", SUSPENDED) + get a =##> \case ("", c, Msg "file description") -> c == bId; _ -> False + where + waitSuspendedOp :: AgentClient -> (AgentClient -> TVar AgentOpState) -> IO () + waitSuspendedOp c op = atomically $ readTVar (op c) >>= checkSTM . opSuspended + suspendedOps :: AgentClient -> IO [Bool] + suspendedOps c = mapM (\op -> opSuspended <$> readTVarIO (op c)) [rcvNetworkOp, sndFilesOp, msgDeliveryOp] + testBatchedSubscriptions :: HasTestEnv => Int -> Int -> (ASrvTransport, AStoreType) -> IO () testBatchedSubscriptions nCreate nDel ps@(t, ASType qsType _) = do (conns, conns') <- withAgentClientsCfgServers2 agentCfg agentCfg initAgentServers2 $ \a b -> do diff --git a/tests/XFTPAgent.hs b/tests/XFTPAgent.hs index e55b9483c..a74bbe444 100644 --- a/tests/XFTPAgent.hs +++ b/tests/XFTPAgent.hs @@ -11,7 +11,7 @@ module XFTPAgent where -import AgentTests.FunctionalAPITests (get, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation) +import AgentTests.FunctionalAPITests (get, nGet, pGet, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation) import Control.Logger.Simple import Control.Monad @@ -19,10 +19,12 @@ import Control.Monad.Except import Control.Monad.IO.Class import qualified Data.ByteString.Char8 as B import qualified Data.ByteString.Lazy as LB +import Data.Either (isLeft) import Data.Int (Int64) import Data.List (find, isSuffixOf) import qualified Data.Map.Strict as M import Data.Maybe (fromJust) +import qualified Data.Set as S import Data.Time.Clock (addUTCTime, getCurrentTime, nominalDay) import Data.Time.Clock.System (getSystemTime, systemSeconds) import SMPAgentClient (agentCfg, initAgentServers, testDB, testDB2, testDB3) @@ -36,11 +38,14 @@ import Simplex.FileTransfer.Transport (XFTPErrorType (AUTH)) import Simplex.FileTransfer.Types (RcvFileId, SndFileId) import Simplex.Messaging.Agent (AgentClient, testProtocolServer, xftpDeleteRcvFile, xftpDeleteSndFileInternal, xftpDeleteSndFileRemote, xftpPrepareReceiveFile, xftpPrepareSendFile, xftpReceiveFile, xftpSendDescription, xftpStartReceiveFile, xftpStartSendFile, xftpStartWorkers) import qualified Simplex.Messaging.Agent as XA -import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..)) +import Simplex.Messaging.Agent.Client (AgentOpState (..), AgentOperation (..), ProtocolTestFailure (..), ProtocolTestStep (..), agentClientStore, beginAgentOperation, endAgentOperation) import Simplex.Messaging.Agent.Env.SQLite (AgentConfig, xftpCfg) import qualified Simplex.Messaging.Agent.Env.SQLite as AEnv import Simplex.Messaging.Agent.Protocol hiding (SFDONE) import qualified Simplex.Messaging.Agent.Protocol as A +import Simplex.Messaging.Agent.Store.Common (withTransaction) +import qualified Simplex.Messaging.Agent.Store.DB as DB +import Simplex.Messaging.Agent.Store.Interface (closeDBStore) import Simplex.Messaging.Client (pattern NRMInteractive) import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Crypto.BBS (bbsKeyGen) @@ -89,6 +94,31 @@ xftpAgentTests = it "if file is deleted on server, should limit retries and continue receiving next file" testXFTPAgentDeleteOnServer it "if file is expired on server, should report error and continue receiving next file" testXFTPAgentExpiredOnServer it "should request additional recipient IDs when number of recipients exceeds maximum per request" testXFTPAgentRequestAdditionalRecipientIDs + it "should complete sending file while agent is suspending" testXFTPAgentSuspendSending + it "should suspend on timeout when file is not sent" $ \_ -> testXFTPAgentSuspendTimeout + it "should not wait for file started while agent is suspending" testXFTPAgentSuspendNewFile + it "should continue sending file when another file is deleted twice while agent is suspending" testXFTPAgentSuspendDeleteTwice + it "should wait for complete file until it is deleted while agent is suspending" testXFTPAgentSuspendCompleteFile + it "should not wait for file deleted from database while agent is suspending" $ \_ -> + testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files") + it "should not wait for file marked deleted while agent is suspending" $ \_ -> + testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET deleted = 1") + it "should not wait for file marked failed while agent is suspending" $ \_ -> + testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET failed = 1") + it "should not wait for file after agent client is closed" $ \_ -> + testXFTPAgentSuspendAfter $ \sndr -> XA.disconnectAgentClient sndr >> XA.resumeAgentClient sndr + it "should not access database when suspending while database is suspended" $ \_ -> testXFTPAgentSuspendDatabaseSuspended + it "should not wait for database when suspending" $ \_ -> testXFTPAgentSuspendDatabaseBusy + it "should not wait for file removed by TTL cleanup while agent is suspending" $ \_ -> testXFTPAgentSuspendExpiredFile + it "should not wait for file resumed at worker start" $ \_ -> testXFTPAgentSuspendResumedFile + it "should stop uploads when sending network is suspended while agent is suspending" testXFTPAgentSuspendSndNetworkSuspended + it "should retry creating file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendPrepareRetry + it "should retry uploading file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendUploadRetry +#if !defined(dbPostgres) + it "should remove counted file when its deletion fails" $ \_ -> testXFTPAgentDeleteSndFileError + it "should keep counted files when database query fails at suspension start" $ \_ -> testXFTPAgentSuspendQueryError +#endif + it "should wait for counted file again after agent is in foreground" $ \_ -> testXFTPAgentSuspendAgain describe "XFTP server test via agent API" $ do it "should pass without basic auth" $ \_ -> testXFTPServerTest Nothing (noAuthSrv testXFTPServer2) `shouldReturn` Right (Just (Right testServerInformation)) let srv1 :: HasTestEnv => XFTPServer @@ -611,6 +641,209 @@ testXFTPAgentSendCleanup = withGlobalLogging logCfgNoLogs $ do doesDirectoryExist prefixPath `shouldReturn` False doesFileExist encPath `shouldReturn` False +testXFTPAgentSuspendSending :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () +testXFTPAgentSuspendSending = withXFTPServer $ do + filePath <- createRandomFile + withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do + xftpStartWorkers sndr (Just senderFiles) + sfId <- xftpSendFile sndr 1 (CF.plain filePath) 2 + liftIO $ XA.suspendAgent sndr 60000000 + liftIO $ timeout 30000000 (sfProgress sndr $ mb 18) `shouldReturn` Just () + ("", sfId', SFDONE _ [_, _]) <- sfGet sndr + liftIO $ do + sfId' `shouldBe` sfId + timeout 300000 (nGet sndr) `shouldReturn` Nothing + xftpDeleteSndFileInternal sndr sfId + timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED) + +testXFTPAgentSuspendTimeout :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendTimeout = withGlobalLogging logCfgNoLogs $ do + filePath <- createRandomFile + withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do + sfId <- runRight $ do + xftpStartWorkers sndr (Just senderFiles) + xftpSendFile sndr 1 (CF.plain filePath) 2 + XA.suspendAgent sndr 500000 + timeout 300000 (nGet sndr) `shouldReturn` Nothing + timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED) + withXFTPServerStoreLogOn $ \_ -> do + timeout 1000000 (sfGet sndr) `shouldReturn` Nothing + XA.foregroundAgent sndr + sfProgress sndr $ mb 18 + ("", sfId', SFDONE _ [_, _]) <- sfGet sndr + sfId' `shouldBe` sfId + +testXFTPAgentSuspendNewFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () +testXFTPAgentSuspendNewFile = withXFTPServer $ do + filePath1 <- createRandomFile' "testfile1" + filePath2 <- createRandomFile' "testfile2" + withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do + xftpStartWorkers sndr (Just senderFiles) + sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1 + liftIO $ XA.suspendAgent sndr 60000000 + sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1 + liftIO $ do + timeout 30000000 (waitSndFileDone sndr sfId1) `shouldReturn` Just () + xftpDeleteSndFileInternal sndr sfId1 + shouldSuspend sndr sfId2 + +testXFTPAgentSuspendDeleteTwice :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () +testXFTPAgentSuspendDeleteTwice = withXFTPServer $ do + filePath1 <- createRandomFile_ (kb 128 :: Integer) "testfile1" + filePath2 <- createRandomFile' "testfile2" + withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do + xftpStartWorkers sndr (Just senderFiles) + sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1 + sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1 + liftIO $ do + XA.suspendAgent sndr 60000000 + xftpDeleteSndFileInternal sndr sfId1 + xftpDeleteSndFileInternal sndr sfId1 + timeout 30000000 (waitSndFileDone sndr sfId2) `shouldReturn` Just () + xftpDeleteSndFileInternal sndr sfId2 + shouldSuspend sndr sfId1 + +testXFTPAgentSuspendCompleteFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () +testXFTPAgentSuspendCompleteFile = withXFTPServer $ + withSndFile agentCfg $ \sndr sfId -> do + timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just () + XA.suspendAgent sndr 60000000 + timeout 300000 (nGet sndr) `shouldReturn` Nothing + xftpDeleteSndFileInternal sndr sfId + timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED) + +testXFTPAgentSuspendAfter :: (HasCallStack, HasTestEnv) => (AgentClient -> IO ()) -> IO () +testXFTPAgentSuspendAfter action = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + action sndr + XA.suspendAgent sndr 60000000 + shouldSuspend sndr sfId + +testXFTPAgentSuspendDatabaseSuspended :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendDatabaseSuspended = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + XA.suspendAgent sndr 0 + withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files") + XA.suspendAgent sndr 60000000 + threadDelay 1000000 + readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.singleton sfId + +testXFTPAgentSuspendDatabaseBusy :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendDatabaseBusy = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + withTransaction (agentClientStore sndr) $ \_ -> do + timeout 1000000 (XA.suspendAgent sndr 500000) `shouldReturn` Just () + timeout 5000000 (atomically $ readTVar (XA.sndNetworkOp sndr) >>= checkSTM . opSuspended) `shouldReturn` Just () + shouldSuspend sndr sfId + +testXFTPAgentSuspendExpiredFile :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendExpiredFile = withGlobalLogging logCfgNoLogs $ do + let cfg = agentCfg {AEnv.initialCleanupDelay = 1000000, AEnv.cleanupInterval = 1000000, AEnv.sndFilesTTL = 1} + withSndFile cfg $ \sndr sfId -> do + XA.suspendAgent sndr 60000000 + timeout 10000000 (waitSuspended sndr sfId) `shouldReturn` Just () + +testXFTPAgentSuspendResumedFile :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendResumedFile = withGlobalLogging logCfgNoLogs $ do + filePath <- createRandomFile_ (kb 128 :: Integer) "testfile" + sfId <- withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight $ do + xftpStartWorkers sndr (Just senderFiles) + xftpSendFile sndr 1 (CF.plain filePath) 1 + withAgent 2 agentCfg initAgentServers testDB $ \sndr -> do + runRight_ $ xftpStartWorkers sndr (Just senderFiles) + threadDelay 500000 + XA.suspendAgent sndr 60000000 + shouldSuspend sndr sfId + +testXFTPAgentSuspendSndNetworkSuspended :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () +testXFTPAgentSuspendSndNetworkSuspended = withXFTPServer $ + withSndFile agentCfg $ \sndr sfId -> do + atomically $ beginAgentOperation sndr AOSndNetwork + XA.suspendAgent sndr 1 + timeout 3000000 (waitSndFileDone sndr sfId) `shouldReturn` Nothing + atomically $ endAgentOperation sndr AOSndNetwork + shouldSuspend sndr sfId + +testXFTPAgentSuspendPrepareRetry :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendPrepareRetry = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + XA.suspendAgent sndr 60000000 + threadDelay 3000000 + withXFTPServerStoreLogOn $ \_ -> do + timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just () + xftpDeleteSndFileInternal sndr sfId + shouldSuspend sndr sfId + +testXFTPAgentSuspendUploadRetry :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendUploadRetry = withGlobalLogging logCfgNoLogs $ do + filePath <- createRandomFile + withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do + sfId <- withXFTPServerStoreLogOn $ \_ -> runRight $ do + xftpStartWorkers sndr (Just senderFiles) + sfId <- xftpSendFile sndr 1 (CF.plain filePath) 1 + ("", sfId', SFPROG _ _) <- sfGet sndr + liftIO $ sfId' `shouldBe` sfId + pure sfId + XA.suspendAgent sndr 60000000 + threadDelay 1000000 + withXFTPServerStoreLogOn $ \_ -> do + timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just () + xftpDeleteSndFileInternal sndr sfId + shouldSuspend sndr sfId + +testXFTPAgentDeleteSndFileError :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentDeleteSndFileError = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + closeDBStore $ agentClientStore sndr + tryAny (xftpDeleteSndFileInternal sndr sfId) >>= (`shouldSatisfy` isLeft) + readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.empty + +testXFTPAgentSuspendQueryError :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendQueryError = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + closeDBStore $ agentClientStore sndr + XA.suspendAgent sndr 60000000 + threadDelay 1000000 + fIds <- readTVarIO $ XA.sndFilesInProgress sndr + XA.foregroundAgent sndr + fIds `shouldBe` S.singleton sfId + +testXFTPAgentSuspendAgain :: (HasCallStack, HasTestEnv) => IO () +testXFTPAgentSuspendAgain = withGlobalLogging logCfgNoLogs $ + withSndFile agentCfg $ \sndr sfId -> do + XA.suspendAgent sndr 500000 + shouldSuspend sndr sfId + XA.foregroundAgent sndr + XA.suspendAgent sndr 60000000 + timeout 1000000 (waitSuspended sndr sfId) `shouldReturn` Nothing + xftpDeleteSndFileInternal sndr sfId + shouldSuspend sndr sfId + +withSndFile :: (HasCallStack, HasTestEnv) => AgentConfig -> (AgentClient -> SndFileId -> IO ()) -> IO () +withSndFile cfg test = do + filePath <- createRandomFile_ (kb 128 :: Integer) "testfile" + withAgent 1 cfg initAgentServers testDB $ \sndr -> do + sfId <- runRight $ do + xftpStartWorkers sndr (Just senderFiles) + xftpSendFile sndr 1 (CF.plain filePath) 1 + test sndr sfId + +shouldSuspend :: HasCallStack => AgentClient -> SndFileId -> Expectation +shouldSuspend sndr sfId = timeout 5000000 (waitSuspended sndr sfId) `shouldReturn` Just () + +waitSndFileDone :: AgentClient -> SndFileId -> IO () +waitSndFileDone sndr sfId = + sfGet sndr >>= \case + ("", sfId', SFDONE _ _) | sfId' == sfId -> pure () + _ -> waitSndFileDone sndr sfId + +waitSuspended :: AgentClient -> SndFileId -> IO () +waitSuspended sndr sfId = + pGet sndr >>= \case + ("", "", AEvt SAENone SUSPENDED) -> pure () + ("", sfId', AEvt SAESndFile _) | sfId' == sfId -> waitSuspended sndr sfId + t -> expectationFailure $ "unexpected event: " <> show t + testXFTPAgentDelete :: (HasCallStack, HasTestEnv) => AFStoreType -> IO () testXFTPAgentDelete = withGlobalLogging logCfgNoLogs . withXFTPServer test where