agent: complete file uploads when the app leaves the foreground (#2010)

* agent: continue XFTP uploads started in foreground while suspending

* tests: verify XFTP upload stage of agent suspension

* docs: plan for XFTP uploads while suspending

* docs: revise plan for XFTP uploads while suspending

* docs: revise plan wording for XFTP uploads while suspending

* agent: exclude ended and stopped XFTP uploads from suspension wait

* tests: verify suspension wait for ended and stopped XFTP uploads

* docs: plan decisions for ended XFTP uploads while suspending

* agent: simplify tryBeginAgentOperation with orElse

* agent: rename upload wait functions by the conditions they check

* tests: verify database check and deleted rows at suspension start

* docs: plan residual case, conditions and names for XFTP suspension

* docs: plan tests for the remaining XFTP suspension paths

* tests: verify the remaining XFTP suspension paths

* docs: update plan line references and the query error test

* tests: foreground agent before checking the query error result

* docs: plan line references after merging master and stage diagrams

* agent: one check of the begun database operation when removing deleted or failed counted files

* agent: suspendAgent returns before the database query of counted files

* tests: closed database store tests excluded from PostgreSQL agent builds

* tests: suspension timeout test waits for the timeout and bounds SUSPENDED

* tests: one setup helper for suspension tests with a small sent file, one for the bounded SUSPENDED wait

* tests: suspendAgent returns while the agent database is in use

* docs: plan states the forked removal at suspension start, uncounted redirects, offline files and the changed tests

* docs: plan links the simplex-chat pull request

* agent: suspend timeout starts when suspendAgent returns, not after the database query of counted files

* docs: plan states the separate removal thread and the TTL cleanup interval
This commit is contained in:
Evgeny @ SimpleX Chat authored and GitHub committed 2026-11-07 15:23:45 +00:00
1 parent 0424f667ee
commit 7ca1f4e508
7 files changed
+569 -25

No files matched your search

@@ -0,0 +1,206 @@
# XFTP uploads while the agent is suspending
On iOS, a file sent shortly before the app left the foreground was delivered only after the app was opened again. Once the agent began to suspend, the next chunk was not started, and the result of the chunk in flight was recorded only in foreground.
While the agent is suspending, it continues the uploads that it started in foreground. Their files are counted by a new operation, `AOSndFiles`, with its own count and set, in a stage between `AORcvNetwork` and `AOMsgDelivery`. The next stage is entered after these files are deleted by the chat, removed by TTL cleanup or `closeAgentClient`, or removed at the start of the suspension when their rows are absent or flagged deleted or failed. At the suspend timeout, the stage of `AOSndNetwork` is entered if the agent is still suspending.
Chat side: https://github.com/simplex-chat/simplex-chat/pull/7966
## Decisions
- Stages of `suspendAgent` with a timeout above 0:
| Order | Operation | Count | At count 0 |
|---|---|---|---|
| 1 | `AORcvNetwork` | received transmission batches in processing by the subscriber | `AOSndFiles` suspended |
| 2 | `AOSndFiles` | counted files | `AOMsgDelivery` suspended |
| 3 | `AOMsgDelivery` | submitted messages not yet taken by delivery workers | `AOSndNetwork` suspended |
| 4 | `AOSndNetwork` | delivery workers and async command workers processing work | `AODatabase` suspended |
| 5 | `AODatabase` | database transactions | `SUSPENDED` sent; agent state `ASSuspended` |
- The actions of "At count 0" are taken only while the agent state is `ASSuspending`.
- `AONtfNetwork` is suspended at the start; the stages are entered independently of its count.
- After the timeout, if the agent is still suspending, stage 4 is entered whatever stage is in progress; stage 5 is entered when the count of stage 4 is 0, as in the table.
```mermaid
stateDiagram-v2
[*] --> RcvNetwork: suspendAgent, timeout above 0
RcvNetwork --> SndFiles: AORcvNetwork count 0
SndFiles --> MsgDelivery: AOSndFiles count 0
MsgDelivery --> SndNetwork: AOMsgDelivery count 0
SndNetwork --> Database: AOSndNetwork count 0
Database --> Suspended: AODatabase count 0, SUSPENDED sent
RcvNetwork --> SndNetwork: timeout
SndFiles --> SndNetwork: timeout
MsgDelivery --> SndNetwork: timeout
Suspended --> [*]
```
- Counted files, the set `sndFilesInProgress`:
- inserted: files started by `xftpStartSendFile'` while the agent state is `ASForeground`;
- removed:
- files deleted by `deleteSndFilesInternal`, after its database work, also when an exception is thrown;
- files expired by TTL cleanup, before their prefix path and row are deleted;
- at the start of a suspension with a timeout above 0, in a thread forked by `suspendAgent`, while `AODatabase` is not suspended: files whose rows are absent or flagged `deleted = 1` or `failed = 1`, found by one query; the suspended flag of `AODatabase` is checked and the operation is begun in one STM transaction; `suspendAgent` returns, and the suspend timeout starts, before this query;
- every file, by `closeAgentClient`.
- kept until the chat deletes them: files with complete or error rows; the file descriptions are sent while these files are counted.
- kept until TTL cleanup, `closeAgentClient` or process exit: counted files completed, or set to an error status, by another process; each suspension in between waits for the suspend timeout.
- kept until they are uploaded or removed: counted files that cannot be uploaded, for example offline; TTL cleanup removes such a file at its first pass after `sndFilesTTL` (1 day) from creation, at most `cleanupInterval` (5 min) later; each suspension in between waits in the files stage until the suspend timeout.
- `deleteSndFilesInternal` is run by `xftpDeleteSndFileInternal`, which is called by the chat after `SFDONE` and `SFERR`.
- Uncounted files:
- files resumed at worker start;
- files started while suspending or suspended;
- among them, the redirect upload of a standalone file, started by the chat on `SFDONE` of its source file (chat `Subscriber.hs:269`, `Internal.hs:497`): its upload stops when the source file is deleted, unless other counted files remain.
- Condition of the send prepare and upload workers, `uploadsAllowed`:
| Agent state | Counted files | `AOSndNetwork` suspended | Condition |
|---|---|---|---|
| `ASForeground` | any | any | true |
| `ASSuspending` | above 0 | no | true |
| `ASSuspending` | 0 | any | false |
| `ASSuspending` | any | yes | false |
| `ASSuspended` | any | any | false |
- While the condition is false, the send prepare and upload workers wait until the agent state is `ASForeground`.
- While the condition is true, uncounted files are uploaded too.
- Receive and delete workers: unchanged.
- `foregroundAgent`: suspended flags cleared; counted files kept.
```
xftpStartSendFile' f ASForeground -> add f
deleteSndFilesInternal fs rows deleted or marked deleted; finally: remove each f
deleteSndFilesExpired f remove f; prefix path and row deleted
suspendAgent, timeout > 0 forked thread: AODatabase not suspended -> begin AODatabase; remove each f whose row is absent, deleted or failed; end AODatabase
closeAgentClient remove each f
add f f not in set -> insert f; count + 1
remove f f in set -> delete f; endAgentOperation AOSndFiles
```
```mermaid
flowchart LR
start["xftpStartSendFile' in ASForeground"] --> add["add f"]
delete["deleteSndFilesInternal, finally"] --> remove["remove f"]
ttl["TTL cleanup"] --> remove
suspend["suspension start, forked thread: row absent, deleted or failed"] --> remove
close["closeAgentClient"] --> remove
remove --> stage["endAgentOperation AOSndFiles"]
```
## Mechanics
- `src/Simplex/Messaging/Agent/Client.hs`:
- `AgentClient` (:379-380), fields created in `newAgentClient` (:550-551):
- `sndFilesOp :: TVar AgentOpState`;
- `sndFilesInProgress :: TVar (Set SndFileId)`.
- `AgentOperation`: `AOSndFiles` (:476).
- `sndFilesOp` added to:
- `agentOpSel` (:483);
- `agentOperations` (:489).
- `closeAgentClient` (:1024): every counted file removed by `removeSndFileInProgress`.
- `endAgentOperation` (:2363-2377): at the end of `AORcvNetwork`, `AOSndFiles` is suspended; at the end of `AOSndFiles`, `suspendMsgDelivery` is called.
- `suspendMsgDelivery` (:2379-2382): `AOMsgDelivery` is suspended; `suspendSendingAndDatabase` is called when its count is 0.
- new:
- `addSndFileInProgress` (:2414-2417);
- `removeSndFileInProgress` (:2419-2422);
- `removeDeletedOrFailedSndFilesInProgress` (:2424-2435): when the set is not empty and `AODatabase` is begun by `tryBeginAgentOperation`, the counted files missing from `getNotDeletedOrFailedSndFileIds` are removed; on a query error, the error is logged and the set is unchanged; under `E.mask`, the begun operation is ended by `E.finally`;
- `tryBeginAgentOperation` (:2446-2447): `beginAgentOperation`, or `False` by `orElse` while the operation is suspended; `True` when begun;
- `waitUntilUploadsAllowed` (:2472-2482), with the condition `uploadsAllowed`.
- `src/Simplex/Messaging/Agent/Store/AgentStore.hs`, new: `getNotDeletedOrFailedSndFileIds` (:3898-3900), the entity IDs of `snd_files` rows with `deleted = 0` and `failed = 0`.
- `src/Simplex/Messaging/Agent.hs`:
- `suspendAgent`:
- stages (:3446-3448): `AORcvNetwork`, `AOSndFiles`, then `suspendMsgDelivery`;
- `removeDeletedOrFailedSndFilesInProgress` (:3451), in a thread forked while the agent is suspending, separate from the timeout thread (:3452-3457); a removal that ends after the timeout holds the `AODatabase` count until it ends, and `SUSPENDED` is sent then.
- `deleteSndFilesExpired` (:3548), local to `cleanupManager`: `removeSndFileInProgress`.
- `src/Simplex/FileTransfer/Agent.hs`:
- `xftpStartSendFile'` (:407): `addSndFileInProgress` when `ASForeground`.
- `assertUploadsAllowed` in place of `assertAgentForeground`:
- `runXFTPSndPrepareWorker` (:424, :487, :507);
- `runXFTPSndWorker` (:528, :549, :560).
- `uploadFileChunk` (:562): `waitUntilUploadsAllowed` in place of `waitUntilForeground`.
- `deleteSndFilesInternal` (:663): `removeSndFileInProgress` in `E.finally`.
- `assertUploadsAllowed` (:749-752): `throwWhenInactive`, then `waitUntilUploadsAllowed`.
## Tests
`tests/XFTPAgent.hs`, "agent XFTP API":
- Helpers:
- `withSndFile` (:822): file of 128 KB; agent 1 with the test's configuration; workers started; the file sent to 1 recipient;
- `shouldSuspend` (:831): `SUSPENDED` within 5 s; events of the file before it are skipped.
- "should complete sending file while agent is suspending" (:644):
- suspend timeout 60 s;
- all `SFPROG` within 30 s, then `SFDONE`;
- events absent for 300 ms;
- after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s.
- "should suspend on timeout when file is not sent" (:659):
- XFTP server down, suspend timeout 0.5 s: events absent for 300 ms, then `SUSPENDED` within 5 s;
- server started: events absent for 1 s;
- after `foregroundAgent`, `SFDONE`.
- "should not wait for file started while agent is suspending" (:676):
- file 1 started, suspend timeout 60 s, file 2 started;
- `SFDONE` of file 1 within 30 s;
- after `xftpDeleteSndFileInternal` of file 1, `shouldSuspend` for file 2.
- "should continue sending file when another file is deleted twice while agent is suspending" (:690):
- file 1 of 128 KB and file 2 of 17 MB started, suspend timeout 60 s;
- `xftpDeleteSndFileInternal` of file 1, twice;
- `SFDONE` of file 2 within 30 s; `SUSPENDED` before it: test failure;
- after `xftpDeleteSndFileInternal` of file 2, `shouldSuspend` for file 1.
- "should wait for complete file until it is deleted while agent is suspending" (:706):
- `withSndFile`; `SFDONE` within 30 s;
- suspend timeout 60 s; events absent for 300 ms;
- after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s.
- `testXFTPAgentSuspendAfter` (:715), for the next four tests:
- XFTP server down; `withSndFile`;
- the test's step;
- suspend timeout 60 s; `shouldSuspend`.
- "should not wait for file deleted from database while agent is suspending" (:102): `DELETE FROM snd_files` through the agent's store.
- "should not wait for file marked deleted while agent is suspending" (:104): `UPDATE snd_files SET deleted = 1` through the agent's store.
- "should not wait for file marked failed while agent is suspending" (:106): `UPDATE snd_files SET failed = 1` through the agent's store.
- "should not wait for file after agent client is closed" (:108): `disconnectAgentClient`, then `resumeAgentClient`.
- "should not access database when suspending while database is suspended" (:722):
- XFTP server down; `withSndFile`;
- suspend timeout 0; the file row deleted by the test;
- suspend timeout 60 s: after 1 s, the set still contains the file.
- "should not wait for database when suspending" (:731):
- XFTP server down; `withSndFile`;
- while the test holds the agent store connection: `suspendAgent` with timeout 0.5 s returns within 1 s; `AOSndNetwork` is suspended by the timeout within 5 s;
- after the test releases the connection, `shouldSuspend`.
- "should not wait for file removed by TTL cleanup while agent is suspending" (:739):
- cleanup after 1 s and every 1 s; `sndFilesTTL` 1 s;
- XFTP server down; `withSndFile`;
- suspend timeout 60 s: `SUSPENDED` within 10 s, after the cleanup removes the expired file; events of the file before it are skipped.
- "should not wait for file resumed at worker start" (:746):
- XFTP server down; file of 128 KB started by agent 1;
- agent 2 on the same database resumes it at `xftpStartWorkers`;
- after 0.5 s, suspend timeout 60 s; `shouldSuspend`.
- "should stop uploads when sending network is suspended while agent is suspending" (:758):
- `withSndFile`; `AOSndNetwork` begun by the test;
- suspend timeout 1 µs: `SFDONE` absent for 3 s;
- after the test ends `AOSndNetwork`, `shouldSuspend`.
- "should retry creating file chunks while agent is suspending" (:767):
- XFTP server down; `withSndFile`; suspend timeout 60 s;
- server started after 3 s: `SFDONE` within 30 s;
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
- "should retry uploading file chunks while agent is suspending" (:777):
- file of 17 MB started; the server is stopped after the first `SFPROG`;
- suspend timeout 60 s; server started after 1 s: `SFDONE` within 30 s;
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
- "should remove counted file when its deletion fails" (:794):
- XFTP server down; `withSndFile`;
- agent store closed: `xftpDeleteSndFileInternal` throws, and the set is empty.
- "should keep counted files when database query fails at suspension start" (:801):
- XFTP server down; `withSndFile`;
- agent store closed; suspend timeout 60 s;
- after 1 s, the set still contains the file; `foregroundAgent` before the comparison.
- The two tests above are excluded from builds with `client_postgres` (`#if !defined(dbPostgres)`, :117-120): a closed PostgreSQL store has no pool connections, and its `withTransaction` waits for one.
- "should wait for counted file again after agent is in foreground" (:811):
- XFTP server down; `withSndFile`;
- suspend timeout 0.5 s; `shouldSuspend`;
- `foregroundAgent`; suspend timeout 60 s: `SUSPENDED` absent for 1 s;
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
`tests/AgentTests/FunctionalAPITests.hs`, "Suspending agent":
- "should send messages queued after file is sent before agent is suspended" (:3177):
- b starts a file, suspend timeout 60 s;
- once `sndFilesOp` is suspended, the `opSuspended` flags of `rcvNetworkOp`, `sndFilesOp`, `msgDeliveryOp`: `[True, True, False]`;
- after `SFDONE`, `sendMessage` and `xftpDeleteSndFileInternal`: `[True, True, True]`;
- `SENT`, then `SUSPENDED` within 5 s;
- a receives the message.
+14 -8
View File
@@ -404,6 +404,7 @@ xftpPrepareSendDescription' c userId (ValidFileDescription fdDirect@FileDescript
xftpStartSendFile' :: AgentClient -> SndFileId -> AM ()
xftpStartSendFile' c sndFileEntityId = do
withStore c (`startPreparedSndFile` sndFileEntityId)
atomically . whenM ((ASForeground ==) <$> readTVar (agentState c)) $ addSndFileInProgress c sndFileEntityId
lift . void $ getXFTPSndWorker True c Nothing
resumeXFTPSndWork :: AgentClient -> Maybe XFTPServer -> AM' ()
@@ -420,7 +421,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
cfg <- asks config
forever $ do
lift $ waitForWork doWork
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
runXFTPOperation cfg
where
runXFTPOperation :: AgentConfig -> AM ()
@@ -483,7 +484,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
SndFileChunkReplica {server} : _ -> Right server
createChunk :: Int -> Maybe Word32 -> SndFileChunk -> AM (ProtocolServer 'PXFTP)
createChunk numRecipients' storageHours ch = do
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
(replica, ProtoServerWithAuth srv _) <- tryCreate
withStore' c $ \db -> createSndFileReplica db ch replica
pure srv
@@ -503,7 +504,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
retryLoop loop triedAllSrvs e = do
flip catchAllErrors (\_ -> pure ()) $ do
when (triedAllSrvs && serverHostError e) $ notify c sndFileEntityId $ SFWARN e
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
loop
createWithNextSrv triedHosts = do
deleted <- withStore' c $ \db -> getSndFileDeleted db sndFileId
@@ -524,7 +525,7 @@ runXFTPSndWorker c srv Worker {doWork} = do
cfg <- asks config
forever $ do
lift $ waitForWork doWork
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
runXFTPOperation cfg
where
runXFTPOperation :: AgentConfig -> AM ()
@@ -545,7 +546,7 @@ runXFTPSndWorker c srv Worker {doWork} = do
when (serverHostError e) $ notify c sndFileEntityId $ SFWARN e
liftIO $ closeXFTPServerClient c userId server digest
withStore' c $ \db -> updateSndChunkReplicaDelay db sndChunkReplicaId replicaDelay
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
loop
retryDone e = do
incXFTPServerStat c userId srv uploadErrs
@@ -556,9 +557,9 @@ runXFTPSndWorker c srv Worker {doWork} = do
fsFilePath <- lift $ toFSFilePath filePath
unlessM (doesFileExist fsFilePath) $ throwE $ FILE NO_FILE
let chunkSpec' = chunkSpec {filePath = fsFilePath} :: XFTPChunkSpec
liftIO $ assertAgentForeground c
liftIO $ assertUploadsAllowed c
agentXFTPUploadChunk c userId chunkDigest replica' chunkSpec'
liftIO $ waitUntilForeground c
liftIO $ waitUntilUploadsAllowed c
sf@SndFile {sndFileEntityId, prefixPath, chunks} <- withStore c $ \db -> do
lockSndFileForUpdate db sndFileId
updateSndChunkReplicaStatus db sndChunkReplicaId SFRSUploaded
@@ -659,7 +660,7 @@ deleteSndFileInternal :: AgentClient -> SndFileId -> AM' ()
deleteSndFileInternal c sndFileEntityId = deleteSndFilesInternal c [sndFileEntityId]
deleteSndFilesInternal :: AgentClient -> [SndFileId] -> AM' ()
deleteSndFilesInternal c sndFileEntityIds = do
deleteSndFilesInternal c sndFileEntityIds = flip E.finally (atomically $ mapM_ (removeSndFileInProgress c) sndFileEntityIds) $ do
sndFiles <- rights <$> withStoreBatch c (\db -> map (fmap (first storeError) . getSndFileByEntityId db) sndFileEntityIds)
let (toDelete, toMarkDeleted) = partition fileComplete sndFiles
workPath <- getXFTPWorkPath
@@ -744,3 +745,8 @@ assertAgentForeground :: AgentClient -> IO ()
assertAgentForeground c = do
throwWhenInactive c
waitUntilForeground c
assertUploadsAllowed :: AgentClient -> IO ()
assertUploadsAllowed c = do
throwWhenInactive c
waitUntilUploadsAllowed c
+11 -8
View File
@@ -3444,15 +3444,17 @@ suspendAgent c@AgentClient {agentState = as} maxDelay = do
writeTVar as ASSuspending
suspendOperation c AONtfNetwork $ pure ()
suspendOperation c AORcvNetwork $
suspendOperation c AOMsgDelivery $
suspendSendingAndDatabase c
suspendOperation c AOSndFiles $
suspendMsgDelivery c
readTVar as
when (state == ASSuspending) . void . forkIO $ do
threadDelay maxDelay
-- liftIO $ putStrLn "suspendAgent after timeout"
atomically . whenSuspending c $ do
-- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase"
suspendSendingAndDatabase c
when (state == ASSuspending) $ do
void . forkIO $ removeDeletedOrFailedSndFilesInProgress c
void . forkIO $ do
threadDelay maxDelay
-- liftIO $ putStrLn "suspendAgent after timeout"
atomically . whenSuspending c $ do
-- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase"
suspendSendingAndDatabase c
execAgentStoreSQL :: AgentClient -> Text -> AE [Text]
execAgentStoreSQL c sql = withAgentEnv c $ withStore' c (`execSQL` sql)
@@ -3544,6 +3546,7 @@ cleanupManager c@AgentClient {subQ} = do
sndFilesTTL <- asks $ sndFilesTTL . config
sndExpired <- withStore' c (`getSndFilesExpired` sndFilesTTL)
forM_ sndExpired $ \(dbId, entId, p) -> flip catchAllErrors (notify entId . SFERR) $ do
atomically $ removeSndFileInProgress c entId
lift . forM_ p $ removePath <=< toFSFilePath
withStore' c (`deleteSndFile'` dbId)
deleteSndFilesDeleted = do
+63 -5
View File
@@ -156,9 +156,14 @@ module Simplex.Messaging.Agent.Client
waitUntilForeground,
waitWhileSuspended,
suspendSendingAndDatabase,
suspendMsgDelivery,
suspendOperation,
notifySuspended,
whenSuspending,
addSndFileInProgress,
removeSndFileInProgress,
removeDeletedOrFailedSndFilesInProgress,
waitUntilUploadsAllowed,
withStore,
withStore',
withStoreBatch,
@@ -235,7 +240,7 @@ import Simplex.FileTransfer.Description (ChunkReplicaId (..), FileDigest (..), k
import Simplex.FileTransfer.Protocol (FileInfo (..), FileResponse)
import Simplex.FileTransfer.Transport (XFTPErrorType (DIGEST), XFTPRcvChunkSpec (..), XFTPVersion)
import qualified Simplex.FileTransfer.Transport as XFTP
import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..))
import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..), SndFileId)
import Simplex.FileTransfer.Util (uniqueCombine)
import Simplex.Messaging.Agent.Env.SQLite
import Simplex.Messaging.Agent.Lock
@@ -371,6 +376,8 @@ data AgentClient = AgentClient
asyncCmdWorkers :: TMap (ConnId, Maybe SMPServer) Worker,
ntfNetworkOp :: TVar AgentOpState,
rcvNetworkOp :: TVar AgentOpState,
sndFilesOp :: TVar AgentOpState,
sndFilesInProgress :: TVar (Set SndFileId),
msgDeliveryOp :: TVar AgentOpState,
sndNetworkOp :: TVar AgentOpState,
databaseOp :: TVar AgentOpState,
@@ -466,19 +473,20 @@ runWorkerAsync Worker {action} work =
where
start = atomically . putTMVar action . Just =<< mkWeakThreadId =<< forkIO work
data AgentOperation = AONtfNetwork | AORcvNetwork | AOMsgDelivery | AOSndNetwork | AODatabase
data AgentOperation = AONtfNetwork | AORcvNetwork | AOSndFiles | AOMsgDelivery | AOSndNetwork | AODatabase
deriving (Eq, Show)
agentOpSel :: AgentOperation -> (AgentClient -> TVar AgentOpState)
agentOpSel = \case
AONtfNetwork -> ntfNetworkOp
AORcvNetwork -> rcvNetworkOp
AOSndFiles -> sndFilesOp
AOMsgDelivery -> msgDeliveryOp
AOSndNetwork -> sndNetworkOp
AODatabase -> databaseOp
agentOperations :: [AgentClient -> TVar AgentOpState]
agentOperations = [ntfNetworkOp, rcvNetworkOp, msgDeliveryOp, sndNetworkOp, databaseOp]
agentOperations = [ntfNetworkOp, rcvNetworkOp, sndFilesOp, msgDeliveryOp, sndNetworkOp, databaseOp]
data AgentOpState = AgentOpState {opSuspended :: !Bool, opsInProgress :: !Int}
@@ -539,6 +547,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf
asyncCmdWorkers <- TM.emptyIO
ntfNetworkOp <- newTVarIO $ AgentOpState False 0
rcvNetworkOp <- newTVarIO $ AgentOpState False 0
sndFilesOp <- newTVarIO $ AgentOpState False 0
sndFilesInProgress <- newTVarIO S.empty
msgDeliveryOp <- newTVarIO $ AgentOpState False 0
sndNetworkOp <- newTVarIO $ AgentOpState False 0
databaseOp <- newTVarIO $ AgentOpState False 0
@@ -583,6 +593,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf
asyncCmdWorkers,
ntfNetworkOp,
rcvNetworkOp,
sndFilesOp,
sndFilesInProgress,
msgDeliveryOp,
sndNetworkOp,
databaseOp,
@@ -1009,6 +1021,7 @@ closeAgentClient c = do
atomically $ SS.clear $ currentSubs c
clear subscrConns
clear getMsgLocks
atomically $ readTVar (sndFilesInProgress c) >>= mapM_ (removeSndFileInProgress c)
where
clearWorkers :: Ord k => (AgentClient -> TMap k a) -> IO (Map k a)
clearWorkers workers = atomically $ swapTVar (workers c) mempty
@@ -2352,8 +2365,10 @@ endAgentOperation :: AgentClient -> AgentOperation -> STM ()
endAgentOperation c op = endOperation c op $ case op of
AONtfNetwork -> pure ()
AORcvNetwork ->
suspendOperation c AOMsgDelivery $
suspendSendingAndDatabase c
suspendOperation c AOSndFiles $
suspendMsgDelivery c
AOSndFiles ->
suspendMsgDelivery c
AOMsgDelivery ->
suspendSendingAndDatabase c
AOSndNetwork ->
@@ -2362,6 +2377,11 @@ endAgentOperation c op = endOperation c op $ case op of
AODatabase ->
notifySuspended c
suspendMsgDelivery :: AgentClient -> STM ()
suspendMsgDelivery c =
suspendOperation c AOMsgDelivery $
suspendSendingAndDatabase c
suspendSendingAndDatabase :: AgentClient -> STM ()
suspendSendingAndDatabase c =
suspendOperation c AOSndNetwork $
@@ -2392,6 +2412,29 @@ whenSuspending :: AgentClient -> STM () -> STM ()
whenSuspending c = whenM ((== ASSuspending) <$> readTVar (agentState c))
{-# INLINE whenSuspending #-}
addSndFileInProgress :: AgentClient -> SndFileId -> STM ()
addSndFileInProgress c fId = do
added <- stateTVar (sndFilesInProgress c) $ \fs -> (S.notMember fId fs, S.insert fId fs)
when added $ modifyTVar' (sndFilesOp c) $ \s -> s {opsInProgress = opsInProgress s + 1}
removeSndFileInProgress :: AgentClient -> SndFileId -> STM ()
removeSndFileInProgress c fId = do
removed <- stateTVar (sndFilesInProgress c) $ \fs -> (S.member fId fs, S.delete fId fs)
when removed $ endAgentOperation c AOSndFiles
removeDeletedOrFailedSndFilesInProgress :: AgentClient -> IO ()
removeDeletedOrFailedSndFilesInProgress c = do
fIds <- readTVarIO $ sndFilesInProgress c
unless (S.null fIds) $
E.mask $ \restore ->
whenM (atomically $ tryBeginAgentOperation c AODatabase) $
restore (removeDeletedOrFailed fIds) `E.finally` atomically (endAgentOperation c AODatabase)
where
removeDeletedOrFailed fIds =
E.tryAny (withTransaction (agentClientStore c) getNotDeletedOrFailedSndFileIds) >>= \case
Right fIds' -> atomically $ mapM_ (removeSndFileInProgress c) $ fIds `S.difference` S.fromList fIds'
Left e -> logError $ "removeDeletedOrFailedSndFilesInProgress error: " <> tshow e
beginAgentOperation :: AgentClient -> AgentOperation -> STM ()
beginAgentOperation c op = do
let opVar = agentOpSel op c
@@ -2401,6 +2444,9 @@ beginAgentOperation c op = do
-- unsafeIOToSTM $ putStrLn $ "beginOperation! " <> show op <> " " <> show (opsInProgress s + 1)
writeTVar opVar $! s {opsInProgress = opsInProgress s + 1}
tryBeginAgentOperation :: AgentClient -> AgentOperation -> STM Bool
tryBeginAgentOperation c op = (True <$ beginAgentOperation c op) `orElse` pure False
agentOperationBracket :: MonadUnliftIO m => AgentClient -> AgentOperation -> (AgentClient -> IO ()) -> m a -> m a
agentOperationBracket c op check action =
E.bracket
@@ -2424,6 +2470,18 @@ waitWhileSuspended c =
suspended :: Monad m => (TVar AgentState -> m AgentState) -> m Bool
suspended rd = (ASSuspended ==) <$> rd (agentState c)
waitUntilUploadsAllowed :: AgentClient -> IO ()
waitUntilUploadsAllowed c = atomically $ unlessM uploadsAllowed retry
where
uploadsAllowed =
readTVar (agentState c) >>= \case
ASForeground -> pure True
ASSuspending -> do
AgentOpState {opsInProgress} <- readTVar $ sndFilesOp c
AgentOpState {opSuspended} <- readTVar $ sndNetworkOp c
pure $ opsInProgress > 0 && not opSuspended
ASSuspended -> pure False
withStore' :: AgentClient -> (DB.Connection -> IO a) -> AM a
withStore' c action = withStore c $ fmap Right . action
{-# INLINE withStore' #-}
@@ -262,6 +262,7 @@ module Simplex.Messaging.Agent.Store.AgentStore
getCleanupSndFilesPrefixPaths,
getCleanupSndFilesDeleted,
getSndFilesExpired,
getNotDeletedOrFailedSndFileIds,
createDeletedSndChunkReplica,
getNextDeletedSndChunkReplica,
updateDeletedSndChunkReplicaDelay,
@@ -3894,6 +3895,10 @@ getSndFilesExpired db ttl = do
|]
(Only cutoffTs)
getNotDeletedOrFailedSndFileIds :: DB.Connection -> IO [SndFileId]
getNotDeletedOrFailedSndFileIds db =
map fromOnly <$> DB.query_ db "SELECT snd_file_entity_id FROM snd_files WHERE deleted = 0 AND failed = 0"
createDeletedSndChunkReplica :: DB.Connection -> UserId -> FileChunkReplica -> FileDigest -> IO ()
createDeletedSndChunkReplica db userId FileChunkReplica {server, replicaId, replicaKey} chunkDigest = do
srvId <- createXFTPServer_ db server
+35 -2
View File
@@ -43,6 +43,7 @@ module AgentTests.FunctionalAPITests
rfGet,
sfGet,
nGet,
pGet,
getInAnyOrder,
(##>),
(=##>),
@@ -90,7 +91,7 @@ import SMPAgentClient
import SMPClient
import Simplex.Messaging.Agent hiding (acceptContact, createConnection, deleteConnection, deleteConnections, getConnShortLink, joinConnection, sendMessage, setConnShortLink, subscribeConnection, suspendConnection)
import qualified Simplex.Messaging.Agent as A
import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork)
import Simplex.Messaging.Agent.Client (AgentOpState (..), ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork)
import Simplex.Messaging.Agent.Env.SQLite (AgentConfig (..), Env (..), InitialAgentServers (..), createAgentStore)
import Simplex.Messaging.Agent.Protocol hiding (CON, CONF, INFO, REQ, SENT)
import qualified Simplex.Messaging.Agent.Protocol as A
@@ -102,6 +103,7 @@ import qualified Simplex.Messaging.Agent.Store.DB as DB
import Simplex.Messaging.Agent.Store.Shared (MigrationConfig (..), MigrationConfirmation (..), MigrationError (..))
import Simplex.Messaging.Client (pattern NRMInteractive, NetworkConfig (..), ProtocolClientConfig (..), TransportSessionMode (..), defaultClientConfig, defaultSMPClientConfig, getProtocolClient, sendSMPMessage)
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Crypto.File (CryptoFile (..))
import Simplex.Messaging.Crypto.Ratchet (InitialKeys (..), PQEncryption (..), PQSupport (..), pattern IKPQOff, pattern IKPQOn, pattern PQEncOff, pattern PQEncOn, pattern PQSupportOff, pattern PQSupportOn)
import qualified Simplex.Messaging.Crypto.Ratchet as CR
import Simplex.Messaging.Encoding
@@ -122,10 +124,11 @@ import Simplex.Messaging.Version (VersionRange (..))
import qualified Simplex.Messaging.Version as V
import Simplex.Messaging.Version.Internal (Version (..))
import System.Directory (copyFile, removeFile, renameFile)
import System.FilePath ((</>))
import Test.Hspec hiding (fit, it)
import UnliftIO
import Util
import XFTPClient (testXFTPServer)
import XFTPClient (testXFTPServer, withXFTPServerThreadOn)
#if defined(dbPostgres)
import Fixtures
@@ -502,6 +505,8 @@ functionalAPITests ps = do
testSuspendingAgentCompleteSending ps
it "should suspend agent on timeout, even if pending messages not sent" $
testSuspendingAgentTimeout ps
it "should send messages queued after file is sent before agent is suspended" $
testSuspendingAgentCompleteSendingFile ps
describe "Batching SMP commands" $ do
-- disable this and enable the following test to run tests with coverage
it "should subscribe to multiple (200) subscriptions with batching" $
@@ -3169,6 +3174,34 @@ testSuspendingAgentTimeout ps = withAgentClients2 $ \a b -> do
("", "", SUSPENDED) <- nGet b
pure ()
testSuspendingAgentCompleteSendingFile :: HasTestEnv => (ASrvTransport, AStoreType) -> IO ()
testSuspendingAgentCompleteSendingFile ps =
withSmpServer ps $ withXFTPServerThreadOn $ \_ -> withAgentClients2 $ \a b -> runRight_ $ do
(aId, bId) <- makeConnection a b
let filePath = senderFiles </> "testfile"
liftIO $ B.writeFile filePath "file content"
xftpStartWorkers b (Just senderFiles)
sfId <- xftpSendFile b 1 (CryptoFile filePath Nothing) 1 Nothing
liftIO $ do
suspendAgent b 60000000
timeout 5000000 (waitSuspendedOp b sndFilesOp) `shouldReturn` Just ()
suspendedOps b `shouldReturn` [True, True, False]
("", _, SFPROG _ _) <- sfGet b
("", sfId', SFDONE {}) <- sfGet b
liftIO $ sfId' `shouldBe` sfId
msgId <- sendMessage b aId SMP.noMsgFlags "file description"
liftIO $ do
xftpDeleteSndFileInternal b sfId
suspendedOps b `shouldReturn` [True, True, True]
get b ##> ("", aId, SENT msgId)
liftIO $ timeout 5000000 (nGet b) `shouldReturn` Just ("", "", SUSPENDED)
get a =##> \case ("", c, Msg "file description") -> c == bId; _ -> False
where
waitSuspendedOp :: AgentClient -> (AgentClient -> TVar AgentOpState) -> IO ()
waitSuspendedOp c op = atomically $ readTVar (op c) >>= checkSTM . opSuspended
suspendedOps :: AgentClient -> IO [Bool]
suspendedOps c = mapM (\op -> opSuspended <$> readTVarIO (op c)) [rcvNetworkOp, sndFilesOp, msgDeliveryOp]
testBatchedSubscriptions :: HasTestEnv => Int -> Int -> (ASrvTransport, AStoreType) -> IO ()
testBatchedSubscriptions nCreate nDel ps@(t, ASType qsType _) = do
(conns, conns') <- withAgentClientsCfgServers2 agentCfg agentCfg initAgentServers2 $ \a b -> do
+235 -2
View File
@@ -11,7 +11,7 @@
module XFTPAgent where
import AgentTests.FunctionalAPITests (get, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation)
import AgentTests.FunctionalAPITests (get, nGet, pGet, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation)
import Control.Logger.Simple
import Control.Monad
@@ -19,10 +19,12 @@ import Control.Monad.Except
import Control.Monad.IO.Class
import qualified Data.ByteString.Char8 as B
import qualified Data.ByteString.Lazy as LB
import Data.Either (isLeft)
import Data.Int (Int64)
import Data.List (find, isSuffixOf)
import qualified Data.Map.Strict as M
import Data.Maybe (fromJust)
import qualified Data.Set as S
import Data.Time.Clock (addUTCTime, getCurrentTime, nominalDay)
import Data.Time.Clock.System (getSystemTime, systemSeconds)
import SMPAgentClient (agentCfg, initAgentServers, testDB, testDB2, testDB3)
@@ -36,11 +38,14 @@ import Simplex.FileTransfer.Transport (XFTPErrorType (AUTH))
import Simplex.FileTransfer.Types (RcvFileId, SndFileId)
import Simplex.Messaging.Agent (AgentClient, testProtocolServer, xftpDeleteRcvFile, xftpDeleteSndFileInternal, xftpDeleteSndFileRemote, xftpPrepareReceiveFile, xftpPrepareSendFile, xftpReceiveFile, xftpSendDescription, xftpStartReceiveFile, xftpStartSendFile, xftpStartWorkers)
import qualified Simplex.Messaging.Agent as XA
import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..))
import Simplex.Messaging.Agent.Client (AgentOpState (..), AgentOperation (..), ProtocolTestFailure (..), ProtocolTestStep (..), agentClientStore, beginAgentOperation, endAgentOperation)
import Simplex.Messaging.Agent.Env.SQLite (AgentConfig, xftpCfg)
import qualified Simplex.Messaging.Agent.Env.SQLite as AEnv
import Simplex.Messaging.Agent.Protocol hiding (SFDONE)
import qualified Simplex.Messaging.Agent.Protocol as A
import Simplex.Messaging.Agent.Store.Common (withTransaction)
import qualified Simplex.Messaging.Agent.Store.DB as DB
import Simplex.Messaging.Agent.Store.Interface (closeDBStore)
import Simplex.Messaging.Client (pattern NRMInteractive)
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Crypto.BBS (bbsKeyGen)
@@ -89,6 +94,31 @@ xftpAgentTests =
it "if file is deleted on server, should limit retries and continue receiving next file" testXFTPAgentDeleteOnServer
it "if file is expired on server, should report error and continue receiving next file" testXFTPAgentExpiredOnServer
it "should request additional recipient IDs when number of recipients exceeds maximum per request" testXFTPAgentRequestAdditionalRecipientIDs
it "should complete sending file while agent is suspending" testXFTPAgentSuspendSending
it "should suspend on timeout when file is not sent" $ \_ -> testXFTPAgentSuspendTimeout
it "should not wait for file started while agent is suspending" testXFTPAgentSuspendNewFile
it "should continue sending file when another file is deleted twice while agent is suspending" testXFTPAgentSuspendDeleteTwice
it "should wait for complete file until it is deleted while agent is suspending" testXFTPAgentSuspendCompleteFile
it "should not wait for file deleted from database while agent is suspending" $ \_ ->
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files")
it "should not wait for file marked deleted while agent is suspending" $ \_ ->
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET deleted = 1")
it "should not wait for file marked failed while agent is suspending" $ \_ ->
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET failed = 1")
it "should not wait for file after agent client is closed" $ \_ ->
testXFTPAgentSuspendAfter $ \sndr -> XA.disconnectAgentClient sndr >> XA.resumeAgentClient sndr
it "should not access database when suspending while database is suspended" $ \_ -> testXFTPAgentSuspendDatabaseSuspended
it "should not wait for database when suspending" $ \_ -> testXFTPAgentSuspendDatabaseBusy
it "should not wait for file removed by TTL cleanup while agent is suspending" $ \_ -> testXFTPAgentSuspendExpiredFile
it "should not wait for file resumed at worker start" $ \_ -> testXFTPAgentSuspendResumedFile
it "should stop uploads when sending network is suspended while agent is suspending" testXFTPAgentSuspendSndNetworkSuspended
it "should retry creating file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendPrepareRetry
it "should retry uploading file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendUploadRetry
#if !defined(dbPostgres)
it "should remove counted file when its deletion fails" $ \_ -> testXFTPAgentDeleteSndFileError
it "should keep counted files when database query fails at suspension start" $ \_ -> testXFTPAgentSuspendQueryError
#endif
it "should wait for counted file again after agent is in foreground" $ \_ -> testXFTPAgentSuspendAgain
describe "XFTP server test via agent API" $ do
it "should pass without basic auth" $ \_ -> testXFTPServerTest Nothing (noAuthSrv testXFTPServer2) `shouldReturn` Right (Just (Right testServerInformation))
let srv1 :: HasTestEnv => XFTPServer
@@ -611,6 +641,209 @@ testXFTPAgentSendCleanup = withGlobalLogging logCfgNoLogs $ do
doesDirectoryExist prefixPath `shouldReturn` False
doesFileExist encPath `shouldReturn` False
testXFTPAgentSuspendSending :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentSuspendSending = withXFTPServer $ do
filePath <- createRandomFile
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
xftpStartWorkers sndr (Just senderFiles)
sfId <- xftpSendFile sndr 1 (CF.plain filePath) 2
liftIO $ XA.suspendAgent sndr 60000000
liftIO $ timeout 30000000 (sfProgress sndr $ mb 18) `shouldReturn` Just ()
("", sfId', SFDONE _ [_, _]) <- sfGet sndr
liftIO $ do
sfId' `shouldBe` sfId
timeout 300000 (nGet sndr) `shouldReturn` Nothing
xftpDeleteSndFileInternal sndr sfId
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
testXFTPAgentSuspendTimeout :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendTimeout = withGlobalLogging logCfgNoLogs $ do
filePath <- createRandomFile
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do
sfId <- runRight $ do
xftpStartWorkers sndr (Just senderFiles)
xftpSendFile sndr 1 (CF.plain filePath) 2
XA.suspendAgent sndr 500000
timeout 300000 (nGet sndr) `shouldReturn` Nothing
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
withXFTPServerStoreLogOn $ \_ -> do
timeout 1000000 (sfGet sndr) `shouldReturn` Nothing
XA.foregroundAgent sndr
sfProgress sndr $ mb 18
("", sfId', SFDONE _ [_, _]) <- sfGet sndr
sfId' `shouldBe` sfId
testXFTPAgentSuspendNewFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentSuspendNewFile = withXFTPServer $ do
filePath1 <- createRandomFile' "testfile1"
filePath2 <- createRandomFile' "testfile2"
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
xftpStartWorkers sndr (Just senderFiles)
sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1
liftIO $ XA.suspendAgent sndr 60000000
sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1
liftIO $ do
timeout 30000000 (waitSndFileDone sndr sfId1) `shouldReturn` Just ()
xftpDeleteSndFileInternal sndr sfId1
shouldSuspend sndr sfId2
testXFTPAgentSuspendDeleteTwice :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentSuspendDeleteTwice = withXFTPServer $ do
filePath1 <- createRandomFile_ (kb 128 :: Integer) "testfile1"
filePath2 <- createRandomFile' "testfile2"
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
xftpStartWorkers sndr (Just senderFiles)
sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1
sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1
liftIO $ do
XA.suspendAgent sndr 60000000
xftpDeleteSndFileInternal sndr sfId1
xftpDeleteSndFileInternal sndr sfId1
timeout 30000000 (waitSndFileDone sndr sfId2) `shouldReturn` Just ()
xftpDeleteSndFileInternal sndr sfId2
shouldSuspend sndr sfId1
testXFTPAgentSuspendCompleteFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentSuspendCompleteFile = withXFTPServer $
withSndFile agentCfg $ \sndr sfId -> do
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
XA.suspendAgent sndr 60000000
timeout 300000 (nGet sndr) `shouldReturn` Nothing
xftpDeleteSndFileInternal sndr sfId
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
testXFTPAgentSuspendAfter :: (HasCallStack, HasTestEnv) => (AgentClient -> IO ()) -> IO ()
testXFTPAgentSuspendAfter action = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
action sndr
XA.suspendAgent sndr 60000000
shouldSuspend sndr sfId
testXFTPAgentSuspendDatabaseSuspended :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendDatabaseSuspended = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
XA.suspendAgent sndr 0
withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files")
XA.suspendAgent sndr 60000000
threadDelay 1000000
readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.singleton sfId
testXFTPAgentSuspendDatabaseBusy :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendDatabaseBusy = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
withTransaction (agentClientStore sndr) $ \_ -> do
timeout 1000000 (XA.suspendAgent sndr 500000) `shouldReturn` Just ()
timeout 5000000 (atomically $ readTVar (XA.sndNetworkOp sndr) >>= checkSTM . opSuspended) `shouldReturn` Just ()
shouldSuspend sndr sfId
testXFTPAgentSuspendExpiredFile :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendExpiredFile = withGlobalLogging logCfgNoLogs $ do
let cfg = agentCfg {AEnv.initialCleanupDelay = 1000000, AEnv.cleanupInterval = 1000000, AEnv.sndFilesTTL = 1}
withSndFile cfg $ \sndr sfId -> do
XA.suspendAgent sndr 60000000
timeout 10000000 (waitSuspended sndr sfId) `shouldReturn` Just ()
testXFTPAgentSuspendResumedFile :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendResumedFile = withGlobalLogging logCfgNoLogs $ do
filePath <- createRandomFile_ (kb 128 :: Integer) "testfile"
sfId <- withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight $ do
xftpStartWorkers sndr (Just senderFiles)
xftpSendFile sndr 1 (CF.plain filePath) 1
withAgent 2 agentCfg initAgentServers testDB $ \sndr -> do
runRight_ $ xftpStartWorkers sndr (Just senderFiles)
threadDelay 500000
XA.suspendAgent sndr 60000000
shouldSuspend sndr sfId
testXFTPAgentSuspendSndNetworkSuspended :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentSuspendSndNetworkSuspended = withXFTPServer $
withSndFile agentCfg $ \sndr sfId -> do
atomically $ beginAgentOperation sndr AOSndNetwork
XA.suspendAgent sndr 1
timeout 3000000 (waitSndFileDone sndr sfId) `shouldReturn` Nothing
atomically $ endAgentOperation sndr AOSndNetwork
shouldSuspend sndr sfId
testXFTPAgentSuspendPrepareRetry :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendPrepareRetry = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
XA.suspendAgent sndr 60000000
threadDelay 3000000
withXFTPServerStoreLogOn $ \_ -> do
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
xftpDeleteSndFileInternal sndr sfId
shouldSuspend sndr sfId
testXFTPAgentSuspendUploadRetry :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendUploadRetry = withGlobalLogging logCfgNoLogs $ do
filePath <- createRandomFile
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do
sfId <- withXFTPServerStoreLogOn $ \_ -> runRight $ do
xftpStartWorkers sndr (Just senderFiles)
sfId <- xftpSendFile sndr 1 (CF.plain filePath) 1
("", sfId', SFPROG _ _) <- sfGet sndr
liftIO $ sfId' `shouldBe` sfId
pure sfId
XA.suspendAgent sndr 60000000
threadDelay 1000000
withXFTPServerStoreLogOn $ \_ -> do
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
xftpDeleteSndFileInternal sndr sfId
shouldSuspend sndr sfId
testXFTPAgentDeleteSndFileError :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentDeleteSndFileError = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
closeDBStore $ agentClientStore sndr
tryAny (xftpDeleteSndFileInternal sndr sfId) >>= (`shouldSatisfy` isLeft)
readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.empty
testXFTPAgentSuspendQueryError :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendQueryError = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
closeDBStore $ agentClientStore sndr
XA.suspendAgent sndr 60000000
threadDelay 1000000
fIds <- readTVarIO $ XA.sndFilesInProgress sndr
XA.foregroundAgent sndr
fIds `shouldBe` S.singleton sfId
testXFTPAgentSuspendAgain :: (HasCallStack, HasTestEnv) => IO ()
testXFTPAgentSuspendAgain = withGlobalLogging logCfgNoLogs $
withSndFile agentCfg $ \sndr sfId -> do
XA.suspendAgent sndr 500000
shouldSuspend sndr sfId
XA.foregroundAgent sndr
XA.suspendAgent sndr 60000000
timeout 1000000 (waitSuspended sndr sfId) `shouldReturn` Nothing
xftpDeleteSndFileInternal sndr sfId
shouldSuspend sndr sfId
withSndFile :: (HasCallStack, HasTestEnv) => AgentConfig -> (AgentClient -> SndFileId -> IO ()) -> IO ()
withSndFile cfg test = do
filePath <- createRandomFile_ (kb 128 :: Integer) "testfile"
withAgent 1 cfg initAgentServers testDB $ \sndr -> do
sfId <- runRight $ do
xftpStartWorkers sndr (Just senderFiles)
xftpSendFile sndr 1 (CF.plain filePath) 1
test sndr sfId
shouldSuspend :: HasCallStack => AgentClient -> SndFileId -> Expectation
shouldSuspend sndr sfId = timeout 5000000 (waitSuspended sndr sfId) `shouldReturn` Just ()
waitSndFileDone :: AgentClient -> SndFileId -> IO ()
waitSndFileDone sndr sfId =
sfGet sndr >>= \case
("", sfId', SFDONE _ _) | sfId' == sfId -> pure ()
_ -> waitSndFileDone sndr sfId
waitSuspended :: AgentClient -> SndFileId -> IO ()
waitSuspended sndr sfId =
pGet sndr >>= \case
("", "", AEvt SAENone SUSPENDED) -> pure ()
("", sfId', AEvt SAESndFile _) | sfId' == sfId -> waitSuspended sndr sfId
t -> expectationFailure $ "unexpected event: " <> show t
testXFTPAgentDelete :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
testXFTPAgentDelete = withGlobalLogging logCfgNoLogs . withXFTPServer test
where