mirror of
https://github.com/simplex-chat/simplexmq.git
synced 2026-11-22 19:58:39 +00:00
agent: complete file uploads when the app leaves the foreground (#2010)
* agent: continue XFTP uploads started in foreground while suspending * tests: verify XFTP upload stage of agent suspension * docs: plan for XFTP uploads while suspending * docs: revise plan for XFTP uploads while suspending * docs: revise plan wording for XFTP uploads while suspending * agent: exclude ended and stopped XFTP uploads from suspension wait * tests: verify suspension wait for ended and stopped XFTP uploads * docs: plan decisions for ended XFTP uploads while suspending * agent: simplify tryBeginAgentOperation with orElse * agent: rename upload wait functions by the conditions they check * tests: verify database check and deleted rows at suspension start * docs: plan residual case, conditions and names for XFTP suspension * docs: plan tests for the remaining XFTP suspension paths * tests: verify the remaining XFTP suspension paths * docs: update plan line references and the query error test * tests: foreground agent before checking the query error result * docs: plan line references after merging master and stage diagrams * agent: one check of the begun database operation when removing deleted or failed counted files * agent: suspendAgent returns before the database query of counted files * tests: closed database store tests excluded from PostgreSQL agent builds * tests: suspension timeout test waits for the timeout and bounds SUSPENDED * tests: one setup helper for suspension tests with a small sent file, one for the bounded SUSPENDED wait * tests: suspendAgent returns while the agent database is in use * docs: plan states the forked removal at suspension start, uncounted redirects, offline files and the changed tests * docs: plan links the simplex-chat pull request * agent: suspend timeout starts when suspendAgent returns, not after the database query of counted files * docs: plan states the separate removal thread and the TTL cleanup interval
This commit is contained in:
1 parent
0424f667ee
commit
7ca1f4e508
7 files changed
+569
-25
No files matched your search
@@ -0,0 +1,206 @@
|
||||
# XFTP uploads while the agent is suspending
|
||||
|
||||
On iOS, a file sent shortly before the app left the foreground was delivered only after the app was opened again. Once the agent began to suspend, the next chunk was not started, and the result of the chunk in flight was recorded only in foreground.
|
||||
While the agent is suspending, it continues the uploads that it started in foreground. Their files are counted by a new operation, `AOSndFiles`, with its own count and set, in a stage between `AORcvNetwork` and `AOMsgDelivery`. The next stage is entered after these files are deleted by the chat, removed by TTL cleanup or `closeAgentClient`, or removed at the start of the suspension when their rows are absent or flagged deleted or failed. At the suspend timeout, the stage of `AOSndNetwork` is entered if the agent is still suspending.
|
||||
|
||||
Chat side: https://github.com/simplex-chat/simplex-chat/pull/7966
|
||||
|
||||
## Decisions
|
||||
|
||||
- Stages of `suspendAgent` with a timeout above 0:
|
||||
|
||||
| Order | Operation | Count | At count 0 |
|
||||
|---|---|---|---|
|
||||
| 1 | `AORcvNetwork` | received transmission batches in processing by the subscriber | `AOSndFiles` suspended |
|
||||
| 2 | `AOSndFiles` | counted files | `AOMsgDelivery` suspended |
|
||||
| 3 | `AOMsgDelivery` | submitted messages not yet taken by delivery workers | `AOSndNetwork` suspended |
|
||||
| 4 | `AOSndNetwork` | delivery workers and async command workers processing work | `AODatabase` suspended |
|
||||
| 5 | `AODatabase` | database transactions | `SUSPENDED` sent; agent state `ASSuspended` |
|
||||
|
||||
- The actions of "At count 0" are taken only while the agent state is `ASSuspending`.
|
||||
- `AONtfNetwork` is suspended at the start; the stages are entered independently of its count.
|
||||
- After the timeout, if the agent is still suspending, stage 4 is entered whatever stage is in progress; stage 5 is entered when the count of stage 4 is 0, as in the table.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> RcvNetwork: suspendAgent, timeout above 0
|
||||
RcvNetwork --> SndFiles: AORcvNetwork count 0
|
||||
SndFiles --> MsgDelivery: AOSndFiles count 0
|
||||
MsgDelivery --> SndNetwork: AOMsgDelivery count 0
|
||||
SndNetwork --> Database: AOSndNetwork count 0
|
||||
Database --> Suspended: AODatabase count 0, SUSPENDED sent
|
||||
RcvNetwork --> SndNetwork: timeout
|
||||
SndFiles --> SndNetwork: timeout
|
||||
MsgDelivery --> SndNetwork: timeout
|
||||
Suspended --> [*]
|
||||
```
|
||||
|
||||
- Counted files, the set `sndFilesInProgress`:
|
||||
- inserted: files started by `xftpStartSendFile'` while the agent state is `ASForeground`;
|
||||
- removed:
|
||||
- files deleted by `deleteSndFilesInternal`, after its database work, also when an exception is thrown;
|
||||
- files expired by TTL cleanup, before their prefix path and row are deleted;
|
||||
- at the start of a suspension with a timeout above 0, in a thread forked by `suspendAgent`, while `AODatabase` is not suspended: files whose rows are absent or flagged `deleted = 1` or `failed = 1`, found by one query; the suspended flag of `AODatabase` is checked and the operation is begun in one STM transaction; `suspendAgent` returns, and the suspend timeout starts, before this query;
|
||||
- every file, by `closeAgentClient`.
|
||||
- kept until the chat deletes them: files with complete or error rows; the file descriptions are sent while these files are counted.
|
||||
- kept until TTL cleanup, `closeAgentClient` or process exit: counted files completed, or set to an error status, by another process; each suspension in between waits for the suspend timeout.
|
||||
- kept until they are uploaded or removed: counted files that cannot be uploaded, for example offline; TTL cleanup removes such a file at its first pass after `sndFilesTTL` (1 day) from creation, at most `cleanupInterval` (5 min) later; each suspension in between waits in the files stage until the suspend timeout.
|
||||
- `deleteSndFilesInternal` is run by `xftpDeleteSndFileInternal`, which is called by the chat after `SFDONE` and `SFERR`.
|
||||
- Uncounted files:
|
||||
- files resumed at worker start;
|
||||
- files started while suspending or suspended;
|
||||
- among them, the redirect upload of a standalone file, started by the chat on `SFDONE` of its source file (chat `Subscriber.hs:269`, `Internal.hs:497`): its upload stops when the source file is deleted, unless other counted files remain.
|
||||
- Condition of the send prepare and upload workers, `uploadsAllowed`:
|
||||
|
||||
| Agent state | Counted files | `AOSndNetwork` suspended | Condition |
|
||||
|---|---|---|---|
|
||||
| `ASForeground` | any | any | true |
|
||||
| `ASSuspending` | above 0 | no | true |
|
||||
| `ASSuspending` | 0 | any | false |
|
||||
| `ASSuspending` | any | yes | false |
|
||||
| `ASSuspended` | any | any | false |
|
||||
|
||||
- While the condition is false, the send prepare and upload workers wait until the agent state is `ASForeground`.
|
||||
- While the condition is true, uncounted files are uploaded too.
|
||||
- Receive and delete workers: unchanged.
|
||||
- `foregroundAgent`: suspended flags cleared; counted files kept.
|
||||
|
||||
```
|
||||
xftpStartSendFile' f ASForeground -> add f
|
||||
deleteSndFilesInternal fs rows deleted or marked deleted; finally: remove each f
|
||||
deleteSndFilesExpired f remove f; prefix path and row deleted
|
||||
suspendAgent, timeout > 0 forked thread: AODatabase not suspended -> begin AODatabase; remove each f whose row is absent, deleted or failed; end AODatabase
|
||||
closeAgentClient remove each f
|
||||
add f f not in set -> insert f; count + 1
|
||||
remove f f in set -> delete f; endAgentOperation AOSndFiles
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
start["xftpStartSendFile' in ASForeground"] --> add["add f"]
|
||||
delete["deleteSndFilesInternal, finally"] --> remove["remove f"]
|
||||
ttl["TTL cleanup"] --> remove
|
||||
suspend["suspension start, forked thread: row absent, deleted or failed"] --> remove
|
||||
close["closeAgentClient"] --> remove
|
||||
remove --> stage["endAgentOperation AOSndFiles"]
|
||||
```
|
||||
|
||||
## Mechanics
|
||||
|
||||
- `src/Simplex/Messaging/Agent/Client.hs`:
|
||||
- `AgentClient` (:379-380), fields created in `newAgentClient` (:550-551):
|
||||
- `sndFilesOp :: TVar AgentOpState`;
|
||||
- `sndFilesInProgress :: TVar (Set SndFileId)`.
|
||||
- `AgentOperation`: `AOSndFiles` (:476).
|
||||
- `sndFilesOp` added to:
|
||||
- `agentOpSel` (:483);
|
||||
- `agentOperations` (:489).
|
||||
- `closeAgentClient` (:1024): every counted file removed by `removeSndFileInProgress`.
|
||||
- `endAgentOperation` (:2363-2377): at the end of `AORcvNetwork`, `AOSndFiles` is suspended; at the end of `AOSndFiles`, `suspendMsgDelivery` is called.
|
||||
- `suspendMsgDelivery` (:2379-2382): `AOMsgDelivery` is suspended; `suspendSendingAndDatabase` is called when its count is 0.
|
||||
- new:
|
||||
- `addSndFileInProgress` (:2414-2417);
|
||||
- `removeSndFileInProgress` (:2419-2422);
|
||||
- `removeDeletedOrFailedSndFilesInProgress` (:2424-2435): when the set is not empty and `AODatabase` is begun by `tryBeginAgentOperation`, the counted files missing from `getNotDeletedOrFailedSndFileIds` are removed; on a query error, the error is logged and the set is unchanged; under `E.mask`, the begun operation is ended by `E.finally`;
|
||||
- `tryBeginAgentOperation` (:2446-2447): `beginAgentOperation`, or `False` by `orElse` while the operation is suspended; `True` when begun;
|
||||
- `waitUntilUploadsAllowed` (:2472-2482), with the condition `uploadsAllowed`.
|
||||
- `src/Simplex/Messaging/Agent/Store/AgentStore.hs`, new: `getNotDeletedOrFailedSndFileIds` (:3898-3900), the entity IDs of `snd_files` rows with `deleted = 0` and `failed = 0`.
|
||||
- `src/Simplex/Messaging/Agent.hs`:
|
||||
- `suspendAgent`:
|
||||
- stages (:3446-3448): `AORcvNetwork`, `AOSndFiles`, then `suspendMsgDelivery`;
|
||||
- `removeDeletedOrFailedSndFilesInProgress` (:3451), in a thread forked while the agent is suspending, separate from the timeout thread (:3452-3457); a removal that ends after the timeout holds the `AODatabase` count until it ends, and `SUSPENDED` is sent then.
|
||||
- `deleteSndFilesExpired` (:3548), local to `cleanupManager`: `removeSndFileInProgress`.
|
||||
- `src/Simplex/FileTransfer/Agent.hs`:
|
||||
- `xftpStartSendFile'` (:407): `addSndFileInProgress` when `ASForeground`.
|
||||
- `assertUploadsAllowed` in place of `assertAgentForeground`:
|
||||
- `runXFTPSndPrepareWorker` (:424, :487, :507);
|
||||
- `runXFTPSndWorker` (:528, :549, :560).
|
||||
- `uploadFileChunk` (:562): `waitUntilUploadsAllowed` in place of `waitUntilForeground`.
|
||||
- `deleteSndFilesInternal` (:663): `removeSndFileInProgress` in `E.finally`.
|
||||
- `assertUploadsAllowed` (:749-752): `throwWhenInactive`, then `waitUntilUploadsAllowed`.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/XFTPAgent.hs`, "agent XFTP API":
|
||||
- Helpers:
|
||||
- `withSndFile` (:822): file of 128 KB; agent 1 with the test's configuration; workers started; the file sent to 1 recipient;
|
||||
- `shouldSuspend` (:831): `SUSPENDED` within 5 s; events of the file before it are skipped.
|
||||
- "should complete sending file while agent is suspending" (:644):
|
||||
- suspend timeout 60 s;
|
||||
- all `SFPROG` within 30 s, then `SFDONE`;
|
||||
- events absent for 300 ms;
|
||||
- after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s.
|
||||
- "should suspend on timeout when file is not sent" (:659):
|
||||
- XFTP server down, suspend timeout 0.5 s: events absent for 300 ms, then `SUSPENDED` within 5 s;
|
||||
- server started: events absent for 1 s;
|
||||
- after `foregroundAgent`, `SFDONE`.
|
||||
- "should not wait for file started while agent is suspending" (:676):
|
||||
- file 1 started, suspend timeout 60 s, file 2 started;
|
||||
- `SFDONE` of file 1 within 30 s;
|
||||
- after `xftpDeleteSndFileInternal` of file 1, `shouldSuspend` for file 2.
|
||||
- "should continue sending file when another file is deleted twice while agent is suspending" (:690):
|
||||
- file 1 of 128 KB and file 2 of 17 MB started, suspend timeout 60 s;
|
||||
- `xftpDeleteSndFileInternal` of file 1, twice;
|
||||
- `SFDONE` of file 2 within 30 s; `SUSPENDED` before it: test failure;
|
||||
- after `xftpDeleteSndFileInternal` of file 2, `shouldSuspend` for file 1.
|
||||
- "should wait for complete file until it is deleted while agent is suspending" (:706):
|
||||
- `withSndFile`; `SFDONE` within 30 s;
|
||||
- suspend timeout 60 s; events absent for 300 ms;
|
||||
- after `xftpDeleteSndFileInternal`, `SUSPENDED` within 5 s.
|
||||
- `testXFTPAgentSuspendAfter` (:715), for the next four tests:
|
||||
- XFTP server down; `withSndFile`;
|
||||
- the test's step;
|
||||
- suspend timeout 60 s; `shouldSuspend`.
|
||||
- "should not wait for file deleted from database while agent is suspending" (:102): `DELETE FROM snd_files` through the agent's store.
|
||||
- "should not wait for file marked deleted while agent is suspending" (:104): `UPDATE snd_files SET deleted = 1` through the agent's store.
|
||||
- "should not wait for file marked failed while agent is suspending" (:106): `UPDATE snd_files SET failed = 1` through the agent's store.
|
||||
- "should not wait for file after agent client is closed" (:108): `disconnectAgentClient`, then `resumeAgentClient`.
|
||||
- "should not access database when suspending while database is suspended" (:722):
|
||||
- XFTP server down; `withSndFile`;
|
||||
- suspend timeout 0; the file row deleted by the test;
|
||||
- suspend timeout 60 s: after 1 s, the set still contains the file.
|
||||
- "should not wait for database when suspending" (:731):
|
||||
- XFTP server down; `withSndFile`;
|
||||
- while the test holds the agent store connection: `suspendAgent` with timeout 0.5 s returns within 1 s; `AOSndNetwork` is suspended by the timeout within 5 s;
|
||||
- after the test releases the connection, `shouldSuspend`.
|
||||
- "should not wait for file removed by TTL cleanup while agent is suspending" (:739):
|
||||
- cleanup after 1 s and every 1 s; `sndFilesTTL` 1 s;
|
||||
- XFTP server down; `withSndFile`;
|
||||
- suspend timeout 60 s: `SUSPENDED` within 10 s, after the cleanup removes the expired file; events of the file before it are skipped.
|
||||
- "should not wait for file resumed at worker start" (:746):
|
||||
- XFTP server down; file of 128 KB started by agent 1;
|
||||
- agent 2 on the same database resumes it at `xftpStartWorkers`;
|
||||
- after 0.5 s, suspend timeout 60 s; `shouldSuspend`.
|
||||
- "should stop uploads when sending network is suspended while agent is suspending" (:758):
|
||||
- `withSndFile`; `AOSndNetwork` begun by the test;
|
||||
- suspend timeout 1 µs: `SFDONE` absent for 3 s;
|
||||
- after the test ends `AOSndNetwork`, `shouldSuspend`.
|
||||
- "should retry creating file chunks while agent is suspending" (:767):
|
||||
- XFTP server down; `withSndFile`; suspend timeout 60 s;
|
||||
- server started after 3 s: `SFDONE` within 30 s;
|
||||
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
|
||||
- "should retry uploading file chunks while agent is suspending" (:777):
|
||||
- file of 17 MB started; the server is stopped after the first `SFPROG`;
|
||||
- suspend timeout 60 s; server started after 1 s: `SFDONE` within 30 s;
|
||||
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
|
||||
- "should remove counted file when its deletion fails" (:794):
|
||||
- XFTP server down; `withSndFile`;
|
||||
- agent store closed: `xftpDeleteSndFileInternal` throws, and the set is empty.
|
||||
- "should keep counted files when database query fails at suspension start" (:801):
|
||||
- XFTP server down; `withSndFile`;
|
||||
- agent store closed; suspend timeout 60 s;
|
||||
- after 1 s, the set still contains the file; `foregroundAgent` before the comparison.
|
||||
- The two tests above are excluded from builds with `client_postgres` (`#if !defined(dbPostgres)`, :117-120): a closed PostgreSQL store has no pool connections, and its `withTransaction` waits for one.
|
||||
- "should wait for counted file again after agent is in foreground" (:811):
|
||||
- XFTP server down; `withSndFile`;
|
||||
- suspend timeout 0.5 s; `shouldSuspend`;
|
||||
- `foregroundAgent`; suspend timeout 60 s: `SUSPENDED` absent for 1 s;
|
||||
- after `xftpDeleteSndFileInternal`, `shouldSuspend`.
|
||||
|
||||
`tests/AgentTests/FunctionalAPITests.hs`, "Suspending agent":
|
||||
- "should send messages queued after file is sent before agent is suspended" (:3177):
|
||||
- b starts a file, suspend timeout 60 s;
|
||||
- once `sndFilesOp` is suspended, the `opSuspended` flags of `rcvNetworkOp`, `sndFilesOp`, `msgDeliveryOp`: `[True, True, False]`;
|
||||
- after `SFDONE`, `sendMessage` and `xftpDeleteSndFileInternal`: `[True, True, True]`;
|
||||
- `SENT`, then `SUSPENDED` within 5 s;
|
||||
- a receives the message.
|
||||
@@ -404,6 +404,7 @@ xftpPrepareSendDescription' c userId (ValidFileDescription fdDirect@FileDescript
|
||||
xftpStartSendFile' :: AgentClient -> SndFileId -> AM ()
|
||||
xftpStartSendFile' c sndFileEntityId = do
|
||||
withStore c (`startPreparedSndFile` sndFileEntityId)
|
||||
atomically . whenM ((ASForeground ==) <$> readTVar (agentState c)) $ addSndFileInProgress c sndFileEntityId
|
||||
lift . void $ getXFTPSndWorker True c Nothing
|
||||
|
||||
resumeXFTPSndWork :: AgentClient -> Maybe XFTPServer -> AM' ()
|
||||
@@ -420,7 +421,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
|
||||
cfg <- asks config
|
||||
forever $ do
|
||||
lift $ waitForWork doWork
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
runXFTPOperation cfg
|
||||
where
|
||||
runXFTPOperation :: AgentConfig -> AM ()
|
||||
@@ -483,7 +484,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
|
||||
SndFileChunkReplica {server} : _ -> Right server
|
||||
createChunk :: Int -> Maybe Word32 -> SndFileChunk -> AM (ProtocolServer 'PXFTP)
|
||||
createChunk numRecipients' storageHours ch = do
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
(replica, ProtoServerWithAuth srv _) <- tryCreate
|
||||
withStore' c $ \db -> createSndFileReplica db ch replica
|
||||
pure srv
|
||||
@@ -503,7 +504,7 @@ runXFTPSndPrepareWorker c Worker {doWork} = do
|
||||
retryLoop loop triedAllSrvs e = do
|
||||
flip catchAllErrors (\_ -> pure ()) $ do
|
||||
when (triedAllSrvs && serverHostError e) $ notify c sndFileEntityId $ SFWARN e
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
loop
|
||||
createWithNextSrv triedHosts = do
|
||||
deleted <- withStore' c $ \db -> getSndFileDeleted db sndFileId
|
||||
@@ -524,7 +525,7 @@ runXFTPSndWorker c srv Worker {doWork} = do
|
||||
cfg <- asks config
|
||||
forever $ do
|
||||
lift $ waitForWork doWork
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
runXFTPOperation cfg
|
||||
where
|
||||
runXFTPOperation :: AgentConfig -> AM ()
|
||||
@@ -545,7 +546,7 @@ runXFTPSndWorker c srv Worker {doWork} = do
|
||||
when (serverHostError e) $ notify c sndFileEntityId $ SFWARN e
|
||||
liftIO $ closeXFTPServerClient c userId server digest
|
||||
withStore' c $ \db -> updateSndChunkReplicaDelay db sndChunkReplicaId replicaDelay
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
loop
|
||||
retryDone e = do
|
||||
incXFTPServerStat c userId srv uploadErrs
|
||||
@@ -556,9 +557,9 @@ runXFTPSndWorker c srv Worker {doWork} = do
|
||||
fsFilePath <- lift $ toFSFilePath filePath
|
||||
unlessM (doesFileExist fsFilePath) $ throwE $ FILE NO_FILE
|
||||
let chunkSpec' = chunkSpec {filePath = fsFilePath} :: XFTPChunkSpec
|
||||
liftIO $ assertAgentForeground c
|
||||
liftIO $ assertUploadsAllowed c
|
||||
agentXFTPUploadChunk c userId chunkDigest replica' chunkSpec'
|
||||
liftIO $ waitUntilForeground c
|
||||
liftIO $ waitUntilUploadsAllowed c
|
||||
sf@SndFile {sndFileEntityId, prefixPath, chunks} <- withStore c $ \db -> do
|
||||
lockSndFileForUpdate db sndFileId
|
||||
updateSndChunkReplicaStatus db sndChunkReplicaId SFRSUploaded
|
||||
@@ -659,7 +660,7 @@ deleteSndFileInternal :: AgentClient -> SndFileId -> AM' ()
|
||||
deleteSndFileInternal c sndFileEntityId = deleteSndFilesInternal c [sndFileEntityId]
|
||||
|
||||
deleteSndFilesInternal :: AgentClient -> [SndFileId] -> AM' ()
|
||||
deleteSndFilesInternal c sndFileEntityIds = do
|
||||
deleteSndFilesInternal c sndFileEntityIds = flip E.finally (atomically $ mapM_ (removeSndFileInProgress c) sndFileEntityIds) $ do
|
||||
sndFiles <- rights <$> withStoreBatch c (\db -> map (fmap (first storeError) . getSndFileByEntityId db) sndFileEntityIds)
|
||||
let (toDelete, toMarkDeleted) = partition fileComplete sndFiles
|
||||
workPath <- getXFTPWorkPath
|
||||
@@ -744,3 +745,8 @@ assertAgentForeground :: AgentClient -> IO ()
|
||||
assertAgentForeground c = do
|
||||
throwWhenInactive c
|
||||
waitUntilForeground c
|
||||
|
||||
assertUploadsAllowed :: AgentClient -> IO ()
|
||||
assertUploadsAllowed c = do
|
||||
throwWhenInactive c
|
||||
waitUntilUploadsAllowed c
|
||||
@@ -3444,15 +3444,17 @@ suspendAgent c@AgentClient {agentState = as} maxDelay = do
|
||||
writeTVar as ASSuspending
|
||||
suspendOperation c AONtfNetwork $ pure ()
|
||||
suspendOperation c AORcvNetwork $
|
||||
suspendOperation c AOMsgDelivery $
|
||||
suspendSendingAndDatabase c
|
||||
suspendOperation c AOSndFiles $
|
||||
suspendMsgDelivery c
|
||||
readTVar as
|
||||
when (state == ASSuspending) . void . forkIO $ do
|
||||
threadDelay maxDelay
|
||||
-- liftIO $ putStrLn "suspendAgent after timeout"
|
||||
atomically . whenSuspending c $ do
|
||||
-- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase"
|
||||
suspendSendingAndDatabase c
|
||||
when (state == ASSuspending) $ do
|
||||
void . forkIO $ removeDeletedOrFailedSndFilesInProgress c
|
||||
void . forkIO $ do
|
||||
threadDelay maxDelay
|
||||
-- liftIO $ putStrLn "suspendAgent after timeout"
|
||||
atomically . whenSuspending c $ do
|
||||
-- unsafeIOToSTM $ putStrLn $ "in timeout: suspendSendingAndDatabase"
|
||||
suspendSendingAndDatabase c
|
||||
|
||||
execAgentStoreSQL :: AgentClient -> Text -> AE [Text]
|
||||
execAgentStoreSQL c sql = withAgentEnv c $ withStore' c (`execSQL` sql)
|
||||
@@ -3544,6 +3546,7 @@ cleanupManager c@AgentClient {subQ} = do
|
||||
sndFilesTTL <- asks $ sndFilesTTL . config
|
||||
sndExpired <- withStore' c (`getSndFilesExpired` sndFilesTTL)
|
||||
forM_ sndExpired $ \(dbId, entId, p) -> flip catchAllErrors (notify entId . SFERR) $ do
|
||||
atomically $ removeSndFileInProgress c entId
|
||||
lift . forM_ p $ removePath <=< toFSFilePath
|
||||
withStore' c (`deleteSndFile'` dbId)
|
||||
deleteSndFilesDeleted = do
|
||||
|
||||
@@ -156,9 +156,14 @@ module Simplex.Messaging.Agent.Client
|
||||
waitUntilForeground,
|
||||
waitWhileSuspended,
|
||||
suspendSendingAndDatabase,
|
||||
suspendMsgDelivery,
|
||||
suspendOperation,
|
||||
notifySuspended,
|
||||
whenSuspending,
|
||||
addSndFileInProgress,
|
||||
removeSndFileInProgress,
|
||||
removeDeletedOrFailedSndFilesInProgress,
|
||||
waitUntilUploadsAllowed,
|
||||
withStore,
|
||||
withStore',
|
||||
withStoreBatch,
|
||||
@@ -235,7 +240,7 @@ import Simplex.FileTransfer.Description (ChunkReplicaId (..), FileDigest (..), k
|
||||
import Simplex.FileTransfer.Protocol (FileInfo (..), FileResponse)
|
||||
import Simplex.FileTransfer.Transport (XFTPErrorType (DIGEST), XFTPRcvChunkSpec (..), XFTPVersion)
|
||||
import qualified Simplex.FileTransfer.Transport as XFTP
|
||||
import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..))
|
||||
import Simplex.FileTransfer.Types (DeletedSndChunkReplica (..), NewSndChunkReplica (..), RcvFileChunkReplica (..), SndFileChunk (..), SndFileChunkReplica (..), SndFileId)
|
||||
import Simplex.FileTransfer.Util (uniqueCombine)
|
||||
import Simplex.Messaging.Agent.Env.SQLite
|
||||
import Simplex.Messaging.Agent.Lock
|
||||
@@ -371,6 +376,8 @@ data AgentClient = AgentClient
|
||||
asyncCmdWorkers :: TMap (ConnId, Maybe SMPServer) Worker,
|
||||
ntfNetworkOp :: TVar AgentOpState,
|
||||
rcvNetworkOp :: TVar AgentOpState,
|
||||
sndFilesOp :: TVar AgentOpState,
|
||||
sndFilesInProgress :: TVar (Set SndFileId),
|
||||
msgDeliveryOp :: TVar AgentOpState,
|
||||
sndNetworkOp :: TVar AgentOpState,
|
||||
databaseOp :: TVar AgentOpState,
|
||||
@@ -466,19 +473,20 @@ runWorkerAsync Worker {action} work =
|
||||
where
|
||||
start = atomically . putTMVar action . Just =<< mkWeakThreadId =<< forkIO work
|
||||
|
||||
data AgentOperation = AONtfNetwork | AORcvNetwork | AOMsgDelivery | AOSndNetwork | AODatabase
|
||||
data AgentOperation = AONtfNetwork | AORcvNetwork | AOSndFiles | AOMsgDelivery | AOSndNetwork | AODatabase
|
||||
deriving (Eq, Show)
|
||||
|
||||
agentOpSel :: AgentOperation -> (AgentClient -> TVar AgentOpState)
|
||||
agentOpSel = \case
|
||||
AONtfNetwork -> ntfNetworkOp
|
||||
AORcvNetwork -> rcvNetworkOp
|
||||
AOSndFiles -> sndFilesOp
|
||||
AOMsgDelivery -> msgDeliveryOp
|
||||
AOSndNetwork -> sndNetworkOp
|
||||
AODatabase -> databaseOp
|
||||
|
||||
agentOperations :: [AgentClient -> TVar AgentOpState]
|
||||
agentOperations = [ntfNetworkOp, rcvNetworkOp, msgDeliveryOp, sndNetworkOp, databaseOp]
|
||||
agentOperations = [ntfNetworkOp, rcvNetworkOp, sndFilesOp, msgDeliveryOp, sndNetworkOp, databaseOp]
|
||||
|
||||
data AgentOpState = AgentOpState {opSuspended :: !Bool, opsInProgress :: !Int}
|
||||
|
||||
@@ -539,6 +547,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf
|
||||
asyncCmdWorkers <- TM.emptyIO
|
||||
ntfNetworkOp <- newTVarIO $ AgentOpState False 0
|
||||
rcvNetworkOp <- newTVarIO $ AgentOpState False 0
|
||||
sndFilesOp <- newTVarIO $ AgentOpState False 0
|
||||
sndFilesInProgress <- newTVarIO S.empty
|
||||
msgDeliveryOp <- newTVarIO $ AgentOpState False 0
|
||||
sndNetworkOp <- newTVarIO $ AgentOpState False 0
|
||||
databaseOp <- newTVarIO $ AgentOpState False 0
|
||||
@@ -583,6 +593,8 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, entitlements, netCf
|
||||
asyncCmdWorkers,
|
||||
ntfNetworkOp,
|
||||
rcvNetworkOp,
|
||||
sndFilesOp,
|
||||
sndFilesInProgress,
|
||||
msgDeliveryOp,
|
||||
sndNetworkOp,
|
||||
databaseOp,
|
||||
@@ -1009,6 +1021,7 @@ closeAgentClient c = do
|
||||
atomically $ SS.clear $ currentSubs c
|
||||
clear subscrConns
|
||||
clear getMsgLocks
|
||||
atomically $ readTVar (sndFilesInProgress c) >>= mapM_ (removeSndFileInProgress c)
|
||||
where
|
||||
clearWorkers :: Ord k => (AgentClient -> TMap k a) -> IO (Map k a)
|
||||
clearWorkers workers = atomically $ swapTVar (workers c) mempty
|
||||
@@ -2352,8 +2365,10 @@ endAgentOperation :: AgentClient -> AgentOperation -> STM ()
|
||||
endAgentOperation c op = endOperation c op $ case op of
|
||||
AONtfNetwork -> pure ()
|
||||
AORcvNetwork ->
|
||||
suspendOperation c AOMsgDelivery $
|
||||
suspendSendingAndDatabase c
|
||||
suspendOperation c AOSndFiles $
|
||||
suspendMsgDelivery c
|
||||
AOSndFiles ->
|
||||
suspendMsgDelivery c
|
||||
AOMsgDelivery ->
|
||||
suspendSendingAndDatabase c
|
||||
AOSndNetwork ->
|
||||
@@ -2362,6 +2377,11 @@ endAgentOperation c op = endOperation c op $ case op of
|
||||
AODatabase ->
|
||||
notifySuspended c
|
||||
|
||||
suspendMsgDelivery :: AgentClient -> STM ()
|
||||
suspendMsgDelivery c =
|
||||
suspendOperation c AOMsgDelivery $
|
||||
suspendSendingAndDatabase c
|
||||
|
||||
suspendSendingAndDatabase :: AgentClient -> STM ()
|
||||
suspendSendingAndDatabase c =
|
||||
suspendOperation c AOSndNetwork $
|
||||
@@ -2392,6 +2412,29 @@ whenSuspending :: AgentClient -> STM () -> STM ()
|
||||
whenSuspending c = whenM ((== ASSuspending) <$> readTVar (agentState c))
|
||||
{-# INLINE whenSuspending #-}
|
||||
|
||||
addSndFileInProgress :: AgentClient -> SndFileId -> STM ()
|
||||
addSndFileInProgress c fId = do
|
||||
added <- stateTVar (sndFilesInProgress c) $ \fs -> (S.notMember fId fs, S.insert fId fs)
|
||||
when added $ modifyTVar' (sndFilesOp c) $ \s -> s {opsInProgress = opsInProgress s + 1}
|
||||
|
||||
removeSndFileInProgress :: AgentClient -> SndFileId -> STM ()
|
||||
removeSndFileInProgress c fId = do
|
||||
removed <- stateTVar (sndFilesInProgress c) $ \fs -> (S.member fId fs, S.delete fId fs)
|
||||
when removed $ endAgentOperation c AOSndFiles
|
||||
|
||||
removeDeletedOrFailedSndFilesInProgress :: AgentClient -> IO ()
|
||||
removeDeletedOrFailedSndFilesInProgress c = do
|
||||
fIds <- readTVarIO $ sndFilesInProgress c
|
||||
unless (S.null fIds) $
|
||||
E.mask $ \restore ->
|
||||
whenM (atomically $ tryBeginAgentOperation c AODatabase) $
|
||||
restore (removeDeletedOrFailed fIds) `E.finally` atomically (endAgentOperation c AODatabase)
|
||||
where
|
||||
removeDeletedOrFailed fIds =
|
||||
E.tryAny (withTransaction (agentClientStore c) getNotDeletedOrFailedSndFileIds) >>= \case
|
||||
Right fIds' -> atomically $ mapM_ (removeSndFileInProgress c) $ fIds `S.difference` S.fromList fIds'
|
||||
Left e -> logError $ "removeDeletedOrFailedSndFilesInProgress error: " <> tshow e
|
||||
|
||||
beginAgentOperation :: AgentClient -> AgentOperation -> STM ()
|
||||
beginAgentOperation c op = do
|
||||
let opVar = agentOpSel op c
|
||||
@@ -2401,6 +2444,9 @@ beginAgentOperation c op = do
|
||||
-- unsafeIOToSTM $ putStrLn $ "beginOperation! " <> show op <> " " <> show (opsInProgress s + 1)
|
||||
writeTVar opVar $! s {opsInProgress = opsInProgress s + 1}
|
||||
|
||||
tryBeginAgentOperation :: AgentClient -> AgentOperation -> STM Bool
|
||||
tryBeginAgentOperation c op = (True <$ beginAgentOperation c op) `orElse` pure False
|
||||
|
||||
agentOperationBracket :: MonadUnliftIO m => AgentClient -> AgentOperation -> (AgentClient -> IO ()) -> m a -> m a
|
||||
agentOperationBracket c op check action =
|
||||
E.bracket
|
||||
@@ -2424,6 +2470,18 @@ waitWhileSuspended c =
|
||||
suspended :: Monad m => (TVar AgentState -> m AgentState) -> m Bool
|
||||
suspended rd = (ASSuspended ==) <$> rd (agentState c)
|
||||
|
||||
waitUntilUploadsAllowed :: AgentClient -> IO ()
|
||||
waitUntilUploadsAllowed c = atomically $ unlessM uploadsAllowed retry
|
||||
where
|
||||
uploadsAllowed =
|
||||
readTVar (agentState c) >>= \case
|
||||
ASForeground -> pure True
|
||||
ASSuspending -> do
|
||||
AgentOpState {opsInProgress} <- readTVar $ sndFilesOp c
|
||||
AgentOpState {opSuspended} <- readTVar $ sndNetworkOp c
|
||||
pure $ opsInProgress > 0 && not opSuspended
|
||||
ASSuspended -> pure False
|
||||
|
||||
withStore' :: AgentClient -> (DB.Connection -> IO a) -> AM a
|
||||
withStore' c action = withStore c $ fmap Right . action
|
||||
{-# INLINE withStore' #-}
|
||||
|
||||
@@ -262,6 +262,7 @@ module Simplex.Messaging.Agent.Store.AgentStore
|
||||
getCleanupSndFilesPrefixPaths,
|
||||
getCleanupSndFilesDeleted,
|
||||
getSndFilesExpired,
|
||||
getNotDeletedOrFailedSndFileIds,
|
||||
createDeletedSndChunkReplica,
|
||||
getNextDeletedSndChunkReplica,
|
||||
updateDeletedSndChunkReplicaDelay,
|
||||
@@ -3894,6 +3895,10 @@ getSndFilesExpired db ttl = do
|
||||
|]
|
||||
(Only cutoffTs)
|
||||
|
||||
getNotDeletedOrFailedSndFileIds :: DB.Connection -> IO [SndFileId]
|
||||
getNotDeletedOrFailedSndFileIds db =
|
||||
map fromOnly <$> DB.query_ db "SELECT snd_file_entity_id FROM snd_files WHERE deleted = 0 AND failed = 0"
|
||||
|
||||
createDeletedSndChunkReplica :: DB.Connection -> UserId -> FileChunkReplica -> FileDigest -> IO ()
|
||||
createDeletedSndChunkReplica db userId FileChunkReplica {server, replicaId, replicaKey} chunkDigest = do
|
||||
srvId <- createXFTPServer_ db server
|
||||
|
||||
@@ -43,6 +43,7 @@ module AgentTests.FunctionalAPITests
|
||||
rfGet,
|
||||
sfGet,
|
||||
nGet,
|
||||
pGet,
|
||||
getInAnyOrder,
|
||||
(##>),
|
||||
(=##>),
|
||||
@@ -90,7 +91,7 @@ import SMPAgentClient
|
||||
import SMPClient
|
||||
import Simplex.Messaging.Agent hiding (acceptContact, createConnection, deleteConnection, deleteConnections, getConnShortLink, joinConnection, sendMessage, setConnShortLink, subscribeConnection, suspendConnection)
|
||||
import qualified Simplex.Messaging.Agent as A
|
||||
import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork)
|
||||
import Simplex.Messaging.Agent.Client (AgentOpState (..), ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), sendAgentMessage, waitForUserNetwork)
|
||||
import Simplex.Messaging.Agent.Env.SQLite (AgentConfig (..), Env (..), InitialAgentServers (..), createAgentStore)
|
||||
import Simplex.Messaging.Agent.Protocol hiding (CON, CONF, INFO, REQ, SENT)
|
||||
import qualified Simplex.Messaging.Agent.Protocol as A
|
||||
@@ -102,6 +103,7 @@ import qualified Simplex.Messaging.Agent.Store.DB as DB
|
||||
import Simplex.Messaging.Agent.Store.Shared (MigrationConfig (..), MigrationConfirmation (..), MigrationError (..))
|
||||
import Simplex.Messaging.Client (pattern NRMInteractive, NetworkConfig (..), ProtocolClientConfig (..), TransportSessionMode (..), defaultClientConfig, defaultSMPClientConfig, getProtocolClient, sendSMPMessage)
|
||||
import qualified Simplex.Messaging.Crypto as C
|
||||
import Simplex.Messaging.Crypto.File (CryptoFile (..))
|
||||
import Simplex.Messaging.Crypto.Ratchet (InitialKeys (..), PQEncryption (..), PQSupport (..), pattern IKPQOff, pattern IKPQOn, pattern PQEncOff, pattern PQEncOn, pattern PQSupportOff, pattern PQSupportOn)
|
||||
import qualified Simplex.Messaging.Crypto.Ratchet as CR
|
||||
import Simplex.Messaging.Encoding
|
||||
@@ -122,10 +124,11 @@ import Simplex.Messaging.Version (VersionRange (..))
|
||||
import qualified Simplex.Messaging.Version as V
|
||||
import Simplex.Messaging.Version.Internal (Version (..))
|
||||
import System.Directory (copyFile, removeFile, renameFile)
|
||||
import System.FilePath ((</>))
|
||||
import Test.Hspec hiding (fit, it)
|
||||
import UnliftIO
|
||||
import Util
|
||||
import XFTPClient (testXFTPServer)
|
||||
import XFTPClient (testXFTPServer, withXFTPServerThreadOn)
|
||||
|
||||
#if defined(dbPostgres)
|
||||
import Fixtures
|
||||
@@ -502,6 +505,8 @@ functionalAPITests ps = do
|
||||
testSuspendingAgentCompleteSending ps
|
||||
it "should suspend agent on timeout, even if pending messages not sent" $
|
||||
testSuspendingAgentTimeout ps
|
||||
it "should send messages queued after file is sent before agent is suspended" $
|
||||
testSuspendingAgentCompleteSendingFile ps
|
||||
describe "Batching SMP commands" $ do
|
||||
-- disable this and enable the following test to run tests with coverage
|
||||
it "should subscribe to multiple (200) subscriptions with batching" $
|
||||
@@ -3169,6 +3174,34 @@ testSuspendingAgentTimeout ps = withAgentClients2 $ \a b -> do
|
||||
("", "", SUSPENDED) <- nGet b
|
||||
pure ()
|
||||
|
||||
testSuspendingAgentCompleteSendingFile :: HasTestEnv => (ASrvTransport, AStoreType) -> IO ()
|
||||
testSuspendingAgentCompleteSendingFile ps =
|
||||
withSmpServer ps $ withXFTPServerThreadOn $ \_ -> withAgentClients2 $ \a b -> runRight_ $ do
|
||||
(aId, bId) <- makeConnection a b
|
||||
let filePath = senderFiles </> "testfile"
|
||||
liftIO $ B.writeFile filePath "file content"
|
||||
xftpStartWorkers b (Just senderFiles)
|
||||
sfId <- xftpSendFile b 1 (CryptoFile filePath Nothing) 1 Nothing
|
||||
liftIO $ do
|
||||
suspendAgent b 60000000
|
||||
timeout 5000000 (waitSuspendedOp b sndFilesOp) `shouldReturn` Just ()
|
||||
suspendedOps b `shouldReturn` [True, True, False]
|
||||
("", _, SFPROG _ _) <- sfGet b
|
||||
("", sfId', SFDONE {}) <- sfGet b
|
||||
liftIO $ sfId' `shouldBe` sfId
|
||||
msgId <- sendMessage b aId SMP.noMsgFlags "file description"
|
||||
liftIO $ do
|
||||
xftpDeleteSndFileInternal b sfId
|
||||
suspendedOps b `shouldReturn` [True, True, True]
|
||||
get b ##> ("", aId, SENT msgId)
|
||||
liftIO $ timeout 5000000 (nGet b) `shouldReturn` Just ("", "", SUSPENDED)
|
||||
get a =##> \case ("", c, Msg "file description") -> c == bId; _ -> False
|
||||
where
|
||||
waitSuspendedOp :: AgentClient -> (AgentClient -> TVar AgentOpState) -> IO ()
|
||||
waitSuspendedOp c op = atomically $ readTVar (op c) >>= checkSTM . opSuspended
|
||||
suspendedOps :: AgentClient -> IO [Bool]
|
||||
suspendedOps c = mapM (\op -> opSuspended <$> readTVarIO (op c)) [rcvNetworkOp, sndFilesOp, msgDeliveryOp]
|
||||
|
||||
testBatchedSubscriptions :: HasTestEnv => Int -> Int -> (ASrvTransport, AStoreType) -> IO ()
|
||||
testBatchedSubscriptions nCreate nDel ps@(t, ASType qsType _) = do
|
||||
(conns, conns') <- withAgentClientsCfgServers2 agentCfg agentCfg initAgentServers2 $ \a b -> do
|
||||
|
||||
+235
-2
@@ -11,7 +11,7 @@
|
||||
|
||||
module XFTPAgent where
|
||||
|
||||
import AgentTests.FunctionalAPITests (get, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation)
|
||||
import AgentTests.FunctionalAPITests (get, nGet, pGet, rfGet, runRight, runRight_, sfGet, withAgent, testServerInformation)
|
||||
|
||||
import Control.Logger.Simple
|
||||
import Control.Monad
|
||||
@@ -19,10 +19,12 @@ import Control.Monad.Except
|
||||
import Control.Monad.IO.Class
|
||||
import qualified Data.ByteString.Char8 as B
|
||||
import qualified Data.ByteString.Lazy as LB
|
||||
import Data.Either (isLeft)
|
||||
import Data.Int (Int64)
|
||||
import Data.List (find, isSuffixOf)
|
||||
import qualified Data.Map.Strict as M
|
||||
import Data.Maybe (fromJust)
|
||||
import qualified Data.Set as S
|
||||
import Data.Time.Clock (addUTCTime, getCurrentTime, nominalDay)
|
||||
import Data.Time.Clock.System (getSystemTime, systemSeconds)
|
||||
import SMPAgentClient (agentCfg, initAgentServers, testDB, testDB2, testDB3)
|
||||
@@ -36,11 +38,14 @@ import Simplex.FileTransfer.Transport (XFTPErrorType (AUTH))
|
||||
import Simplex.FileTransfer.Types (RcvFileId, SndFileId)
|
||||
import Simplex.Messaging.Agent (AgentClient, testProtocolServer, xftpDeleteRcvFile, xftpDeleteSndFileInternal, xftpDeleteSndFileRemote, xftpPrepareReceiveFile, xftpPrepareSendFile, xftpReceiveFile, xftpSendDescription, xftpStartReceiveFile, xftpStartSendFile, xftpStartWorkers)
|
||||
import qualified Simplex.Messaging.Agent as XA
|
||||
import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..))
|
||||
import Simplex.Messaging.Agent.Client (AgentOpState (..), AgentOperation (..), ProtocolTestFailure (..), ProtocolTestStep (..), agentClientStore, beginAgentOperation, endAgentOperation)
|
||||
import Simplex.Messaging.Agent.Env.SQLite (AgentConfig, xftpCfg)
|
||||
import qualified Simplex.Messaging.Agent.Env.SQLite as AEnv
|
||||
import Simplex.Messaging.Agent.Protocol hiding (SFDONE)
|
||||
import qualified Simplex.Messaging.Agent.Protocol as A
|
||||
import Simplex.Messaging.Agent.Store.Common (withTransaction)
|
||||
import qualified Simplex.Messaging.Agent.Store.DB as DB
|
||||
import Simplex.Messaging.Agent.Store.Interface (closeDBStore)
|
||||
import Simplex.Messaging.Client (pattern NRMInteractive)
|
||||
import qualified Simplex.Messaging.Crypto as C
|
||||
import Simplex.Messaging.Crypto.BBS (bbsKeyGen)
|
||||
@@ -89,6 +94,31 @@ xftpAgentTests =
|
||||
it "if file is deleted on server, should limit retries and continue receiving next file" testXFTPAgentDeleteOnServer
|
||||
it "if file is expired on server, should report error and continue receiving next file" testXFTPAgentExpiredOnServer
|
||||
it "should request additional recipient IDs when number of recipients exceeds maximum per request" testXFTPAgentRequestAdditionalRecipientIDs
|
||||
it "should complete sending file while agent is suspending" testXFTPAgentSuspendSending
|
||||
it "should suspend on timeout when file is not sent" $ \_ -> testXFTPAgentSuspendTimeout
|
||||
it "should not wait for file started while agent is suspending" testXFTPAgentSuspendNewFile
|
||||
it "should continue sending file when another file is deleted twice while agent is suspending" testXFTPAgentSuspendDeleteTwice
|
||||
it "should wait for complete file until it is deleted while agent is suspending" testXFTPAgentSuspendCompleteFile
|
||||
it "should not wait for file deleted from database while agent is suspending" $ \_ ->
|
||||
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files")
|
||||
it "should not wait for file marked deleted while agent is suspending" $ \_ ->
|
||||
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET deleted = 1")
|
||||
it "should not wait for file marked failed while agent is suspending" $ \_ ->
|
||||
testXFTPAgentSuspendAfter $ \sndr -> withTransaction (agentClientStore sndr) (`DB.execute_` "UPDATE snd_files SET failed = 1")
|
||||
it "should not wait for file after agent client is closed" $ \_ ->
|
||||
testXFTPAgentSuspendAfter $ \sndr -> XA.disconnectAgentClient sndr >> XA.resumeAgentClient sndr
|
||||
it "should not access database when suspending while database is suspended" $ \_ -> testXFTPAgentSuspendDatabaseSuspended
|
||||
it "should not wait for database when suspending" $ \_ -> testXFTPAgentSuspendDatabaseBusy
|
||||
it "should not wait for file removed by TTL cleanup while agent is suspending" $ \_ -> testXFTPAgentSuspendExpiredFile
|
||||
it "should not wait for file resumed at worker start" $ \_ -> testXFTPAgentSuspendResumedFile
|
||||
it "should stop uploads when sending network is suspended while agent is suspending" testXFTPAgentSuspendSndNetworkSuspended
|
||||
it "should retry creating file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendPrepareRetry
|
||||
it "should retry uploading file chunks while agent is suspending" $ \_ -> testXFTPAgentSuspendUploadRetry
|
||||
#if !defined(dbPostgres)
|
||||
it "should remove counted file when its deletion fails" $ \_ -> testXFTPAgentDeleteSndFileError
|
||||
it "should keep counted files when database query fails at suspension start" $ \_ -> testXFTPAgentSuspendQueryError
|
||||
#endif
|
||||
it "should wait for counted file again after agent is in foreground" $ \_ -> testXFTPAgentSuspendAgain
|
||||
describe "XFTP server test via agent API" $ do
|
||||
it "should pass without basic auth" $ \_ -> testXFTPServerTest Nothing (noAuthSrv testXFTPServer2) `shouldReturn` Right (Just (Right testServerInformation))
|
||||
let srv1 :: HasTestEnv => XFTPServer
|
||||
@@ -611,6 +641,209 @@ testXFTPAgentSendCleanup = withGlobalLogging logCfgNoLogs $ do
|
||||
doesDirectoryExist prefixPath `shouldReturn` False
|
||||
doesFileExist encPath `shouldReturn` False
|
||||
|
||||
testXFTPAgentSuspendSending :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentSuspendSending = withXFTPServer $ do
|
||||
filePath <- createRandomFile
|
||||
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
sfId <- xftpSendFile sndr 1 (CF.plain filePath) 2
|
||||
liftIO $ XA.suspendAgent sndr 60000000
|
||||
liftIO $ timeout 30000000 (sfProgress sndr $ mb 18) `shouldReturn` Just ()
|
||||
("", sfId', SFDONE _ [_, _]) <- sfGet sndr
|
||||
liftIO $ do
|
||||
sfId' `shouldBe` sfId
|
||||
timeout 300000 (nGet sndr) `shouldReturn` Nothing
|
||||
xftpDeleteSndFileInternal sndr sfId
|
||||
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
|
||||
|
||||
testXFTPAgentSuspendTimeout :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendTimeout = withGlobalLogging logCfgNoLogs $ do
|
||||
filePath <- createRandomFile
|
||||
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do
|
||||
sfId <- runRight $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
xftpSendFile sndr 1 (CF.plain filePath) 2
|
||||
XA.suspendAgent sndr 500000
|
||||
timeout 300000 (nGet sndr) `shouldReturn` Nothing
|
||||
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
|
||||
withXFTPServerStoreLogOn $ \_ -> do
|
||||
timeout 1000000 (sfGet sndr) `shouldReturn` Nothing
|
||||
XA.foregroundAgent sndr
|
||||
sfProgress sndr $ mb 18
|
||||
("", sfId', SFDONE _ [_, _]) <- sfGet sndr
|
||||
sfId' `shouldBe` sfId
|
||||
|
||||
testXFTPAgentSuspendNewFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentSuspendNewFile = withXFTPServer $ do
|
||||
filePath1 <- createRandomFile' "testfile1"
|
||||
filePath2 <- createRandomFile' "testfile2"
|
||||
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1
|
||||
liftIO $ XA.suspendAgent sndr 60000000
|
||||
sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1
|
||||
liftIO $ do
|
||||
timeout 30000000 (waitSndFileDone sndr sfId1) `shouldReturn` Just ()
|
||||
xftpDeleteSndFileInternal sndr sfId1
|
||||
shouldSuspend sndr sfId2
|
||||
|
||||
testXFTPAgentSuspendDeleteTwice :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentSuspendDeleteTwice = withXFTPServer $ do
|
||||
filePath1 <- createRandomFile_ (kb 128 :: Integer) "testfile1"
|
||||
filePath2 <- createRandomFile' "testfile2"
|
||||
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight_ $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
sfId1 <- xftpSendFile sndr 1 (CF.plain filePath1) 1
|
||||
sfId2 <- xftpSendFile sndr 1 (CF.plain filePath2) 1
|
||||
liftIO $ do
|
||||
XA.suspendAgent sndr 60000000
|
||||
xftpDeleteSndFileInternal sndr sfId1
|
||||
xftpDeleteSndFileInternal sndr sfId1
|
||||
timeout 30000000 (waitSndFileDone sndr sfId2) `shouldReturn` Just ()
|
||||
xftpDeleteSndFileInternal sndr sfId2
|
||||
shouldSuspend sndr sfId1
|
||||
|
||||
testXFTPAgentSuspendCompleteFile :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentSuspendCompleteFile = withXFTPServer $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
|
||||
XA.suspendAgent sndr 60000000
|
||||
timeout 300000 (nGet sndr) `shouldReturn` Nothing
|
||||
xftpDeleteSndFileInternal sndr sfId
|
||||
timeout 5000000 (nGet sndr) `shouldReturn` Just ("", "", SUSPENDED)
|
||||
|
||||
testXFTPAgentSuspendAfter :: (HasCallStack, HasTestEnv) => (AgentClient -> IO ()) -> IO ()
|
||||
testXFTPAgentSuspendAfter action = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
action sndr
|
||||
XA.suspendAgent sndr 60000000
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentSuspendDatabaseSuspended :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendDatabaseSuspended = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
XA.suspendAgent sndr 0
|
||||
withTransaction (agentClientStore sndr) (`DB.execute_` "DELETE FROM snd_files")
|
||||
XA.suspendAgent sndr 60000000
|
||||
threadDelay 1000000
|
||||
readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.singleton sfId
|
||||
|
||||
testXFTPAgentSuspendDatabaseBusy :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendDatabaseBusy = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
withTransaction (agentClientStore sndr) $ \_ -> do
|
||||
timeout 1000000 (XA.suspendAgent sndr 500000) `shouldReturn` Just ()
|
||||
timeout 5000000 (atomically $ readTVar (XA.sndNetworkOp sndr) >>= checkSTM . opSuspended) `shouldReturn` Just ()
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentSuspendExpiredFile :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendExpiredFile = withGlobalLogging logCfgNoLogs $ do
|
||||
let cfg = agentCfg {AEnv.initialCleanupDelay = 1000000, AEnv.cleanupInterval = 1000000, AEnv.sndFilesTTL = 1}
|
||||
withSndFile cfg $ \sndr sfId -> do
|
||||
XA.suspendAgent sndr 60000000
|
||||
timeout 10000000 (waitSuspended sndr sfId) `shouldReturn` Just ()
|
||||
|
||||
testXFTPAgentSuspendResumedFile :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendResumedFile = withGlobalLogging logCfgNoLogs $ do
|
||||
filePath <- createRandomFile_ (kb 128 :: Integer) "testfile"
|
||||
sfId <- withAgent 1 agentCfg initAgentServers testDB $ \sndr -> runRight $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
xftpSendFile sndr 1 (CF.plain filePath) 1
|
||||
withAgent 2 agentCfg initAgentServers testDB $ \sndr -> do
|
||||
runRight_ $ xftpStartWorkers sndr (Just senderFiles)
|
||||
threadDelay 500000
|
||||
XA.suspendAgent sndr 60000000
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentSuspendSndNetworkSuspended :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentSuspendSndNetworkSuspended = withXFTPServer $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
atomically $ beginAgentOperation sndr AOSndNetwork
|
||||
XA.suspendAgent sndr 1
|
||||
timeout 3000000 (waitSndFileDone sndr sfId) `shouldReturn` Nothing
|
||||
atomically $ endAgentOperation sndr AOSndNetwork
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentSuspendPrepareRetry :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendPrepareRetry = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
XA.suspendAgent sndr 60000000
|
||||
threadDelay 3000000
|
||||
withXFTPServerStoreLogOn $ \_ -> do
|
||||
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
|
||||
xftpDeleteSndFileInternal sndr sfId
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentSuspendUploadRetry :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendUploadRetry = withGlobalLogging logCfgNoLogs $ do
|
||||
filePath <- createRandomFile
|
||||
withAgent 1 agentCfg initAgentServers testDB $ \sndr -> do
|
||||
sfId <- withXFTPServerStoreLogOn $ \_ -> runRight $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
sfId <- xftpSendFile sndr 1 (CF.plain filePath) 1
|
||||
("", sfId', SFPROG _ _) <- sfGet sndr
|
||||
liftIO $ sfId' `shouldBe` sfId
|
||||
pure sfId
|
||||
XA.suspendAgent sndr 60000000
|
||||
threadDelay 1000000
|
||||
withXFTPServerStoreLogOn $ \_ -> do
|
||||
timeout 30000000 (waitSndFileDone sndr sfId) `shouldReturn` Just ()
|
||||
xftpDeleteSndFileInternal sndr sfId
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
testXFTPAgentDeleteSndFileError :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentDeleteSndFileError = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
closeDBStore $ agentClientStore sndr
|
||||
tryAny (xftpDeleteSndFileInternal sndr sfId) >>= (`shouldSatisfy` isLeft)
|
||||
readTVarIO (XA.sndFilesInProgress sndr) `shouldReturn` S.empty
|
||||
|
||||
testXFTPAgentSuspendQueryError :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendQueryError = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
closeDBStore $ agentClientStore sndr
|
||||
XA.suspendAgent sndr 60000000
|
||||
threadDelay 1000000
|
||||
fIds <- readTVarIO $ XA.sndFilesInProgress sndr
|
||||
XA.foregroundAgent sndr
|
||||
fIds `shouldBe` S.singleton sfId
|
||||
|
||||
testXFTPAgentSuspendAgain :: (HasCallStack, HasTestEnv) => IO ()
|
||||
testXFTPAgentSuspendAgain = withGlobalLogging logCfgNoLogs $
|
||||
withSndFile agentCfg $ \sndr sfId -> do
|
||||
XA.suspendAgent sndr 500000
|
||||
shouldSuspend sndr sfId
|
||||
XA.foregroundAgent sndr
|
||||
XA.suspendAgent sndr 60000000
|
||||
timeout 1000000 (waitSuspended sndr sfId) `shouldReturn` Nothing
|
||||
xftpDeleteSndFileInternal sndr sfId
|
||||
shouldSuspend sndr sfId
|
||||
|
||||
withSndFile :: (HasCallStack, HasTestEnv) => AgentConfig -> (AgentClient -> SndFileId -> IO ()) -> IO ()
|
||||
withSndFile cfg test = do
|
||||
filePath <- createRandomFile_ (kb 128 :: Integer) "testfile"
|
||||
withAgent 1 cfg initAgentServers testDB $ \sndr -> do
|
||||
sfId <- runRight $ do
|
||||
xftpStartWorkers sndr (Just senderFiles)
|
||||
xftpSendFile sndr 1 (CF.plain filePath) 1
|
||||
test sndr sfId
|
||||
|
||||
shouldSuspend :: HasCallStack => AgentClient -> SndFileId -> Expectation
|
||||
shouldSuspend sndr sfId = timeout 5000000 (waitSuspended sndr sfId) `shouldReturn` Just ()
|
||||
|
||||
waitSndFileDone :: AgentClient -> SndFileId -> IO ()
|
||||
waitSndFileDone sndr sfId =
|
||||
sfGet sndr >>= \case
|
||||
("", sfId', SFDONE _ _) | sfId' == sfId -> pure ()
|
||||
_ -> waitSndFileDone sndr sfId
|
||||
|
||||
waitSuspended :: AgentClient -> SndFileId -> IO ()
|
||||
waitSuspended sndr sfId =
|
||||
pGet sndr >>= \case
|
||||
("", "", AEvt SAENone SUSPENDED) -> pure ()
|
||||
("", sfId', AEvt SAESndFile _) | sfId' == sfId -> waitSuspended sndr sfId
|
||||
t -> expectationFailure $ "unexpected event: " <> show t
|
||||
|
||||
testXFTPAgentDelete :: (HasCallStack, HasTestEnv) => AFStoreType -> IO ()
|
||||
testXFTPAgentDelete = withGlobalLogging logCfgNoLogs . withXFTPServer test
|
||||
where
|
||||
|
||||
Reference in new issue
Block a user