Commit Graph
292 Commits
Author SHA1 Message Date
Evgeny @ SimpleX Chat 09a6309520 xftp: prevent stuck network IO after interrupted transfers on GHC 8.10 (#2070)
* xftp: prevent stuck network IO after interrupted transfers on GHC 8.10

* xftp: apply uninterruptible chunk file close only on GHC 8.10
2026-11-14 09:14:57 +00:00
Evgeny @ SimpleX Chat 73f734927f agent: read work items in one snapshot, report deleted files as NO_FILE (#2060)
* agent: skip work items deleted by a concurrent transaction

* agent: retry a work item batch whose items were all deleted, test skipping

* tests: skip work items deleted after their IDs were read

* agent: revert skipping of work items deleted after their ID query

* agent: lock work items and their connections in ID queries on PostgreSQL

* xftp: report a file deleted during transfer as NO_FILE, not INTERNAL

* tests: delete work items during worker reads and file transfers

* agent: retry worker reads aborted by PostgreSQL deadlock detection

* xftp: report NO_FILE when a file is deleted before its chunks are saved

* tests: lock tests on readers without item locks, deterministic NO_FILE tests

* agent: keep the SQLite text of work item ID queries unchanged

* tests: resume the upload in a new agent with the server down

* agent: retry transactions and batches aborted by PostgreSQL deadlocks

* tests: lock tests for six ID queries, NO_FILE when adding recipients

* agent: read work items in one snapshot, retry serialization failures

* xftp: report NO_FILE after a failed write when the file is deleted

* tests: work item reads with concurrent deletions on PostgreSQL

* tests: start the prepared file read by the prepare worker test

* agent: export transaction retry for chat work item reads

* agent: avoid shadowing STM retry in the transaction retry loop

* tests: wait for the lock held by the test's own transaction

* agent: retry read conflicts on the same database connection

* tests: retry bound of work item reads on PostgreSQL

* tests: qualify the agent INTERNAL error in retry bound tests

* tests: check retried reads use one connection and report the state

* tests: match the SQLSTATE field in retry bound tests

* tests: failed work item read is marked failed on PostgreSQL

* xftp: apply the deleted-file handler in infix form

* xftp: apply the deleted-file handler before multi-line writes

* agent: remove the unused read committed isolation level
2026-11-14 00:02:25 +00:00
Evgeny @ SimpleX Chat 239290ca98 xftp: limit received files to the size of the sent file (#2067)
* tests: decrypt chunk files into the original file

* xftp: limit received files to the size of the sent file
2026-11-13 20:43:43 +00:00
Evgeny @ SimpleX Chat 99d84f7350 servers: write metrics atomically, log periodic task errors (#2061)
* servers: prevent partial reads of metrics files

* tests: compare metrics time with shouldNotReturn

* servers: log metrics file errors instead of stopping

* servers, tests: test metrics written while read and after a failed write

* tests: retry failed tests outside the exception handler, so timeouts end them

* servers: log metrics collection errors instead of stopping

* servers: log errors of periodic tasks instead of stopping

* servers: count expired files after deletion, read stats counts each period
2026-11-13 10:25:41 +00:00
Evgeny @ SimpleX Chat f9f3bc9208 xftp: prevent work directory conflicts of concurrent file transfers (#2062)
* tests: cover XFTP files prepared concurrently

* xftp: create unique work directories for concurrent file transfers

* xftp: test whether a candidate path is taken

* xftp: compute candidate names apart from checking and creating paths

* xftp: create unique directories in a separate function

* xftp: share the candidate path between unique names and directories
2026-11-13 08:39:49 +00:00
Evgeny @ SimpleX Chat 91bfba321b lib: evaluate the new TVar state in stateTVar updates (#2031)
* lib: evaluate random generator state when it is updated

* lib: evaluate the new state in all stateTVar updates
2026-11-07 23:09:51 +00:00
Evgeny @ SimpleX Chat db998ca19f xftp: limit the redirect file of a description to 4 MB (#2026) 2026-11-07 16:29:06 +00:00
Evgeny @ SimpleX Chat 30d9808f16 xftp: receive files with many chunks, always report receive errors (#2025)
* xftp: receive files with many chunks, always report receive errors

* xftp: hash received file chunks one at a time
2026-11-07 15:57:09 +00:00
Evgeny @ SimpleX Chat 7ca1f4e508 agent: complete file uploads when the app leaves the foreground (#2010)
* agent: continue XFTP uploads started in foreground while suspending

* tests: verify XFTP upload stage of agent suspension

* docs: plan for XFTP uploads while suspending

* docs: revise plan for XFTP uploads while suspending

* docs: revise plan wording for XFTP uploads while suspending

* agent: exclude ended and stopped XFTP uploads from suspension wait

* tests: verify suspension wait for ended and stopped XFTP uploads

* docs: plan decisions for ended XFTP uploads while suspending

* agent: simplify tryBeginAgentOperation with orElse

* agent: rename upload wait functions by the conditions they check

* tests: verify database check and deleted rows at suspension start

* docs: plan residual case, conditions and names for XFTP suspension

* docs: plan tests for the remaining XFTP suspension paths

* tests: verify the remaining XFTP suspension paths

* docs: update plan line references and the query error test

* tests: foreground agent before checking the query error result

* docs: plan line references after merging master and stage diagrams

* agent: one check of the begun database operation when removing deleted or failed counted files

* agent: suspendAgent returns before the database query of counted files

* tests: closed database store tests excluded from PostgreSQL agent builds

* tests: suspension timeout test waits for the timeout and bounds SUSPENDED

* tests: one setup helper for suspension tests with a small sent file, one for the bounded SUSPENDED wait

* tests: suspendAgent returns while the agent database is in use

* docs: plan states the forked removal at suspension start, uncounted redirects, offline files and the changed tests

* docs: plan links the simplex-chat pull request

* agent: suspend timeout starts when suspendAgent returns, not after the database query of counted files

* docs: plan states the separate removal thread and the TTL cleanup interval
2026-11-07 15:23:45 +00:00
Evgeny @ SimpleX Chat 6cfb7cf924 xftp: reject file descriptions with very long numbers (#2018)
* xftp: reject file descriptions with very long numbers

* xftp: simplify the long number guard condition
2026-11-06 22:29:34 +00:00
Evgeny @ SimpleX Chat 026d27d082 xftp: send and receive files through servers before v6.5.0 (#2021)
* xftp: test decoding handshake of servers before v6.5.0

* xftp: send and receive files through servers before v6.5.0

* xftp: refuse server handshake with invalid signature
2026-11-06 18:00:36 +00:00
Evgeny @ SimpleX Chat 32f61fd104 xftp: limit chunk size in received file descriptions (#2001) 2026-11-05 07:48:35 +00:00
Evgeny @ SimpleX Chat c0c416daf0 agent, ntf server: add lookupOrInsertIO, use it for locks and stats (#1989)
* agent, ntf server: add lookupOrInsertIO, use it for locks and stats

* agent, server: use lookupOrInsertIO for map entries

* agent: use lookupOrInsertIO for server stats, keep GET lock in one transaction

* smp server: update getSubscription comment for lookupOrInsertIO result

* agent, server: keep original function names, reduce diff

* agent, server: simplify incServerStat and getMessage

* agent, ntf server: stat under whenPendingSub, lookupOrInsertIO in addPendingSubs

* smp server, session: use lookupOrInsertIO directly, drop the newSub alias

* agent, server: fromEither, remove unused incXFTPServerStat'

* util: hide UnliftIO fromEither where Util fromEither is used

* agent, server: use either id id, remove fromEither

* agent, server: keep isPendingSub, reword service delivery comment
2026-10-26 20:51:24 +00:00
Evgeny @ SimpleX Chat 195df0d094 xftp server: release storage of interrupted uploads (#1977)
* xftp server: release storage of interrupted uploads

* xftp server: release storage of interrupted file deletion
2026-10-26 11:12:25 +00:00
Evgeny @ SimpleX Chat 3818284007 xftp: close connection after failed handshake (#1974) 2026-10-25 13:53:56 +00:00
Evgeny @ SimpleX Chat 568cd2c68c xftp: prohibit YAML aliases, limit file description replicas (#1965)
* xftp: limit description replicas, no YAML aliases

* inline limits

* xftp: derive file description chunk limit from maximum file size

* xftp: allow descriptions of 64 GB files at redundancy 4
2026-10-25 07:32:32 +00:00
Evgeny @ SimpleX Chat 11673c2450 agent, server: one thread per worker; close sockets in reader threads (#1972)
## Agent

- Workers: a worker started by concurrent calls runs in one thread. `runWorkerAsync` put back a stale "not running" value after another call had taken the running one, and a worker restart wrote the same value, so a second thread executed the same commands (for example, a confirmation was sent twice and `CON` was emitted twice), or a caller blocked forever.
- Delivery receipts that fail with AUTH are deleted without an error event.
- Notification credentials are stored per receive queue rather than per connection.
- The protocol client fails pending requests when its connection ends.
- XFTP: a failure to remove a received file's temporary data is logged and no longer fails the worker; server test files have random names.

## Server

- An inactive SMP client session ends without its connection being closed by the expiration thread.
- An inactive HTTP2 session is ended by killing its thread rather than closing its TLS connection from another thread.
- Static files: the timeout handler kills the connection thread and leaves the socket close to that thread.

## Dependencies

- warp: simplex-chat/wai@63667cb5 (simplex-chat/wai#1). The timeout manager kills the connection thread without closing its socket.

## CI and tests

- Tests run on a RAM disk on Linux.
- Test HTTP2 clients are closed before their TLS connections; test temporary directories are cleared between suites.
2026-10-25 07:24:36 +00:00
EvgenyandEvgeny @ SimpleX Chat 9d83c0378e tests: execute in parallel, agent: fix race condition with files and when switching queue (#1932)
* tests: execute in parallel

* fix test

* fix postgresql build

* fix test

* fix test

* fix some tests

* fix tests

* fix tests

* fix test

* update schema

* fix test

* update schemas

* diff

* fix race

* fixes

* update schemas

* fix tests

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-10-15 14:10:04 +01:00
EvgenyandEvgeny @ SimpleX Chat 8d81f775b1 smp protocol: require fixed size corr ID when used as nonce, and min size msg ID to use as nonce (#1925)
* smp protocol: require fixed size corr ID when used as nonce, and min size msg ID to use as nonce

* simpler

* fixes

* randomMsgId

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-10-08 09:32:11 +01:00
EvgenyandEvgeny @ SimpleX Chat c0a1b7bd2d agent: fail database transaction instantly when database is closed, prevent race conditions in command processing, fix race when uploading files (#1885)
* agent: fail database transaction instantly when database is closed

* update direct-sqlcipher

* update direct-sqlcipher, fixes

* autoincrement command ID

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-10-06 13:01:55 +01:00
EvgenyandEvgeny @ SimpleX Chat 0e1ecdcff6 xftp server: validate paths in store log (#1894)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-09-30 09:36:45 +01:00
EvgenyandEvgeny @ SimpleX Chat 35f1b145a7 xftp: fix race when sending the file (#1879)
* xftp: fix race when sending the file

* simpler test

* fix tests

* test

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-09-25 11:59:06 +01:00
EvgenyandEvgeny @ SimpleX Chat 0b97e025ce xftp server: disable ack command (#1871)
* xftp server: disable ack

* fix tests

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-09-12 10:06:09 +01:00
EvgenyandEvgeny @ SimpleX Chat a00e225d74 xftp: hash of the shared part of file description (#1865)
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-09-12 08:30:13 +01:00
EvgenyandEvgeny @ SimpleX Chat 1876357c7f xftp server: support storage time and BBS proofs of badge credential to extend it (#1856)
* xftp server: support storage time and BBS proofs of badge credential to extend it

* update

* types

* implementation

* update

* remove permanent and FTTL

* refactor

* refactor

* refactor

* simplify

* simplify

* simpler

* remove comments

* add file expiration time to agent event

* add test of upload with entitlement

* update schema

* fix header and test

* fix store log encoding, add xftp store log tests

* send entitlement proof in handshake

* type, query

* more fixes

* test entitlement keys

* update

* fixes

* fix

* comment

* remove log

* simpler

* ignore entitlement proofs in browser requests

* remove condition

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-09-01 12:59:00 +01:00
79d1b48252 feat: add server public information handling in XFTP protocol (#1846)
* feat: add server public information handling in XFTP protocol

* test

---------

Co-authored-by: sh <github.shum@liber.li>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-08-31 17:31:57 +01:00
spaced4ndy c377f2c1b1 xftp: prevent sender from writing files outside the destination folder (#1850) 2026-08-12 18:39:03 +01:00
EvgenyandEd Asriyan 0e13d46a2a smp-server: add serverInfoBytes to THandleParams and related functions (#1836)
* smp-server: add serverInfoBytes to THandleParams and related functions

* server information

* fix test

---------

Co-authored-by: Ed Asriyan <service.github@asriyan.me>
2026-08-02 11:45:04 +01:00
Evgeny c2eb680535 clients, servers: drop support of SMP protocol versions before 01/2025 (#1837)
* servers: drop support of versions before 01/2025

* fix agent tests

* fix ntf tests

* fix test

* remove code for non-batched transmissions

* fix batching test

* fix ntf

* comment
2026-07-31 09:43:19 +01:00
EvgenyandPaul Bottinelli b2bdade380 fix: ignore pending XFTP files in storage accounting (#1814)
* fix: ignore pending XFTP files in storage accounting

* style

---------

Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-21 13:27:28 +01:00
shandPaul Bottinelli 8a1b5608bf xftp-cli: add deprecation notice (#1799)
Co-authored-by: Paul Bottinelli <paul.bottinelli@trailofbits.com>
2026-06-16 10:07:13 +01:00
f0b7a4be73 messaging services (#1667)
* smp server: messaging services (#1565)

* smp server: refactor message delivery to always respond SOK to subscriptions

* refactor ntf subscribe

* cancel subscription thread and reduce service subscription count when queue is deleted

* subscribe rcv service, deliver sent messages to subscribed service

* subscribe rcv service to messages (TODO delivery on subscription)

* WIP

* efficient initial delivery of messages to subscribed service

* test: delivery to client with service certificate

* test: upgrade/downgrade to/from service subscriptions

* remove service association from agent API, add per-user flag to use the service

* agent client (WIP)

* service certificates in the client

* rfc about drift detection, and SALL to mark end of message delivery

* fix test

* fix test

* add function for postgresql message storage

* update migration

* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1668)

* servers: maintain xor-hash of all associated queue IDs in PostgreSQL (#1615)

* ntf server: maintain xor-hash of all associated queue IDs via PostgreSQL triggers

* smp server: xor hash with triggers

* fix sql and using pgcrypto extension in tests

* track counts and hashes in smp/ntf servers via triggers, smp server stats for service subscription, update SMP protocol to pass expected count and hash in SSUB/NSSUB commands

* agent migrations with functions/triggers

* remove agent triggers

* try tracking service subs in the agent (WIP, does not compile)

* Revert "try tracking service subs in the agent (WIP, does not compile)"

This reverts commit 59e908100d.

* comment

* agent database triggers

* service subscriptions in the client

* test / fix client services

* update schema

* fix postgres migration

* update schema

* move schema test to the end

* use static function with SQLite to avoid dynamic wrapper

* agent: fail when per-connection transport isolation is used with services (#1670)

* agent: service subscription events (#1671)

* agent: use server keyhash when loading service record

* agent: process queue/service associations with delayed subscription results

* agent: service subscription events

* agent: finalize initial service subscriptions, remove associations on service ID changes (#1672)

* agent: remove service/queue associations when service ID changes

* agent: check that service ID in NEW response matches session ID in transport session

* agent subscription WIP

* test

* comment

* enable tests

* update queries

* agent: option to add SQLite aggregates to DB connection  (#1673)

* agent: add build_relations_vector function to sqlite

* update aggregate

* use static aggregate

* remove relations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* add test, treat BAD_SERVICE as temp error, only remove queue associations on service errors

* add packZipWith for backward compatibility with GHC 8.10.7

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* servers: service stats and logging, allow services without option (removed), report errors during service message delivery, remove threads when service subscription ended (#1676)

* smp server: always allow services without option

* smp server: maintain IDs hash in session subscription states

* smp server: service message delivery error handling

* ntf server: log subscription count and hash differences

* smp server: remove delivery threads when service subscription ended/client disconnected

* agent: remove service queue association when service ID changed, process ENDS event, test migrating to/from service (#1677)

* agent: remove service queue association when service ID changed

* agent: process ENDS event

* agent: send service subscription error event

* agent: test migrating to/from service subscriptions, fixes

* agent: always remove service when disabled, fix service subscriptions

* ntf server: use different client certs for each SMP server, remove support for store log (#1681)

* ntf server: remove support for store log

* ntf server: use different client certificates for each SMP server

* smp protocol: fix encoding for SOKS/ENDS responses (#1683)

* agent: create user with option to enable client service (#1684)

* agent: create user with option to enable client service

* handle HTTP2 errors

* do not catch async exceptions

* agent: minor fixes

* docs: update protocol (#1705)

* docs: agent threat model

* update protocol docs

* update RFCs (#1730)

* update RFCs

* update

* update overview

* update terminology

* original language in threat model

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* docs: fix minor issues in protocols

* docs: add e2e encrypted message wire encoding to PQDR spec

* docs: add missing encodings and other protocol corrections

* docs: move implemented rfcs

* smp: service fixes (#1737)

* smp: deliver service subscription to correct client

* tests: more resilient to concurrency

* optimize PostgreSQL query

* fix service re-association after server "downgrade"

* correctly handle service removed from server (and ID changed)

* remove unused

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* prometheus: fix metrics names (#1747)

* test: rcv service re-association on restart (#1746)

* agent: correct log message

* docs: update whitepaper

* smp: fix messaging client service issues (#1751)

* services: fix minor issues

* fix accounting for subscribed service queues, add prometheus stats

* fix uncorrelated subquery

* fix potential race condition when inserting service defensively, as it is also prevented by how client is created

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* agent: refactor cleanup if no pending subs (#1757)

* smp server: batch processing of subscription messages (#1753)

* smp server: batch processing of subscription messages

* refactor

* empty line

* fix

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* smp: batch queue association updates on subscriptions (#1760)

* smp: batch queue association updates on subscriptions

* refactor to fused batching

* simpler

* batch assoc functions

* clean up

* fix

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>

* agent: use primary key index in setRcvServiceAssocs (#1783)

* agent: use primary key index in setRcvServiceAssocs

Previous WHERE rcv_id = ? did not match the (host, port, rcv_id)
primary key prefix and fell back to a table scan via
idx_rcv_queues_client_notice_id. With ~390k rows per queue, each
update in a 1350-row batch scanned the whole table, yielding ~290s
per batch and a multi-hour rcv-services migration.

* agent: pass SMPServer explicitly to setRcvServiceAssocs

Avoid extracting host/port from the first queue inside setRcvServiceAssocs.
The caller already has SMPServer in scope (from tSess) and the call chain
is short, so threading it through is simpler than inspecting the list.
Removes the empty-list guard from setRcvServiceAssocs (it remains in
processRcvServiceAssocs).

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-05-21 14:14:03 +01:00
Evgeny 21f4597dad xftp: backwards compatible file header decoding (#1768) 2026-05-01 13:32:19 +01:00
sh 1e1f897c79 core: use = as INI key-value separator (#1767)
* core: use = as INI key-value separator

* core: update docker entrypoints for = INI separator

* core: update INI separator in README and test scripts
2026-04-20 09:22:14 +01:00
sh 8833e5c1b5 xftp-server: support postgresql backend (#1755)
* xftp: add PostgreSQL backend design spec

* update doc

* adjust styling

* add implementation plan

* refactor: move usedStorage from FileStore to XFTPEnv

* refactor: add getUsedStorage, getFileCount, expiredFiles store functions

* refactor: change file store operations from STM to IO

* refactor: extract FileStoreClass typeclass, move STM impl to Store.STM

* refactor: make XFTPEnv and server polymorphic over FileStoreClass

* feat: add PostgreSQL store skeleton with schema migration

* feat: implement PostgresFileStore operations

* feat: add PostgreSQL INI config, store dispatch, startup validation

* feat: add database import/export CLI commands

* test: add PostgreSQL backend tests

* fix: map ForeignKeyViolation to AUTH in addRecipient

When a file is concurrently deleted while addRecipient runs, the FK
constraint on recipients.sender_id raises ForeignKeyViolation. Previously
this propagated as INTERNAL; now it returns AUTH (file not found).

* fix: only decrement usedStorage for uploaded files on expiration

expireServerFiles unconditionally subtracted file_size from usedStorage
for every expired file, including files that were never uploaded (no
file_path). Since reserve only increments usedStorage during upload,
expiring never-uploaded files caused usedStorage to drift negative.

* fix: handle setFilePath error in receiveServerFile

setFilePath result was discarded with void. If it failed (file deleted
concurrently, or double-upload where file_path IS NULL guard rejected
the second write), the server still reported FROk, incremented stats,
and left usedStorage permanently inflated. Now the error is checked:
on failure, reserved storage is released and AUTH is returned.

* fix: escape double quotes in COPY CSV status field

The status field (e.g. "blocked,reason=spam,notice={...}") is quoted in
CSV for COPY protocol, but embedded double quotes from BlockingInfo
notice (JSON) were not escaped. This could break CSV parsing during
import. Now double quotes are escaped as "" per CSV spec.

* fix: reject upload to blocked file in Postgres setFilePath

In Postgres mode, getFile returns a snapshot TVar for fileStatus. If a
file is blocked between getFile and setFilePath, the stale status check
passes but the upload should be rejected. Added status = 'active' to
the UPDATE WHERE clause so blocked files cannot receive uploads.

* fix: add CHECK constraint on file_size > 0

Prevents negative or zero file_size values at the database level.
Without this, corrupted data from import or direct DB access could
cause incorrect storage accounting (getUsedStorage sums file_size,
and expiredFiles casts to Word32 which wraps negative values).

* fix: check for existing data before database import

importFileStore now checks if the target database already contains
files and aborts with an error. Previously, importing into a non-empty
database would fail mid-COPY on duplicate primary keys, leaving the
database in a partially imported state.

* fix: clean up disk file when setFilePath fails in receiveServerFile

When setFilePath fails (file deleted or blocked concurrently, or
duplicate upload), the uploaded file was left orphaned on disk with
no DB record pointing to it. Now the file is removed on failure,
matching the cleanup in the receiveChunk error path.

* fix: check storeAction result in deleteOrBlockServerFile_

The store action result (deleteFile/blockFile) was discarded with void.
If the DB row was already deleted by a concurrent operation, the
function still decremented usedStorage, causing drift. Now the error
propagates via ExceptT, skipping the usedStorage adjustment.

* fix: check deleteFile result in expireServerFiles

deleteFile result was discarded with void. If a concurrent delete
already removed the file, deleteFile returned AUTH but usedStorage
was still decremented — causing double-decrement drift. Now the
usedStorage adjustment and filesExpired stat only run on success.

* refactor: merge STM store into Store.hs, parameterize server tests

- Move STMFileStore and its FileStoreClass instance from Store/STM.hs
  back into Store.hs — the separate file was unnecessary indirection
  for the always-present default implementation.

- Parameterize xftpFileTests over store backend using HSpec SpecWith
  pattern (following SMP's serverTests approach). The same 11 tests
  now run against both memory and PostgreSQL backends via a bracket
  parameter, eliminating all *Pg test duplicates.

- Extract shared run* functions (runTestFileChunkDeliveryAddRecipients,
  runTestWrongChunkSize, runTestFileChunkExpiration, runTestFileStorageQuota)
  from inlined test bodies.

* refactor: clean up per good-code review

- Remove internal helpers from Postgres.hs export list (withDB, withDB',
  handleDuplicate, assertUpdated, withLog are not imported externally)
- Replace local isNothing_ with Data.Maybe.isNothing in Env.hs
- Consolidate duplicate/unused imports in XFTPStoreTests.hs
- Add file_path IS NULL and status guards to STM setFilePath, matching
  the Postgres implementation semantics

* test: parameterize XFTP server, agent and CLI tests over store backend

- xftpTest/xftpTest2/xftpTest4/xftpTestN now take XFTPTestBracket as
  first argument, enabling the same test to run against both memory
  and PostgreSQL backends.

- xftpFileTests (server tests), xftpAgentFileTests (agent tests), and
  xftpCLIFileTests (CLI tests) are SpecWith-parameterized suites that
  receive the bracket from HSpec's before combinator.

- Test.hs runs each parameterized suite twice: once with
  xftpMemoryBracket, once with xftpPostgresBracket (CPP-guarded).

- STM-specific tests (store log restore/replay) stay in memory-only
  xftpAgentTests. SNI/CORS tests stay in memory-only xftpServerTests.

* refactor: remove dead test wrappers after parameterization

Remove old non-parameterized test wrapper functions that were
superseded by the store-backend-parameterized test suites.
All test bodies (run* and _ functions) are preserved and called
from the parameterized specs. Clean up unused imports.

* feat: add manual tests and guide

* refactor: merge file_size CHECK into initial migration

* refactor: extract rowToFileRec shared by getFile sender/recipient paths

* refactor: parameterize XFTPServerConfig over store type

Embed XFTPStoreConfig s as serverStoreCfg field, matching SMP's
ServerConfig. runXFTPServer and newXFTPServerEnv now take a single
XFTPServerConfig s. Restore verifyCmd local helper structure.

* refactor: minimize diff in tests

Restore xftpServerTests and xftpAgentTests bodies to match master
byte-for-byte (only type signatures change for XFTPTestBracket
parameterization); inline the runTestXXX helpers that were split
on this branch.

* refactor: restore getFile position to match master

* refactor: rename withSTMFile back to withFile

* refactor: close store log inside closeFileStore for STM backend

Move STM store log close responsibility into closeFileStore to
match PostgresFileStore, removing the asymmetry where only PG's
close was self-contained.

STMFileStore holds the log in a TVar populated by newXFTPServerEnv
after readWriteFileStore; stopServer no longer needs the explicit
withFileLog closeStoreLog call. Writes still go through XFTPEnv.storeLog
via withFileLog (unchanged).

* refactor: rename XFTPTestBracket to XFTPTestServer

* fix: move file_size check from PG schema to store log import

* refactor: use SQL-standard type names in XFTP schema

* perf: batch expired file deletions with deleteFiles

* refactor: stream export instead of loading recipients into memory

* refactor: parameterize XFTP store with FSType singleton dispatch

* refactor: minimize diff per review feedback

* refactor: use types over strings, deduplicate parser

* refactor: always parse database store type, fail at startup

* fix compilation without postgresql

* refactor: always parse database store type, fail at startup
2026-04-16 09:06:04 +01:00
shandEvgeny Poberezkin 437cdde4a5 xftp: add web page for server information (#1724)
* xftp: add web page for server information

* web: rename XFTP.Web to XFTPWeb, remove XFTP subdirectory

* refactor(xftp): remove storage quota from web page

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-09 08:44:28 +00:00
EvgenyandEvgeny @ SimpleX Chat f3408d9bb6 explicit exports (#1719)
* explicit exports

* more empty exports

* add exports

* reorder

* use correct ControlProtocol type for xftp router

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-02 17:34:01 +00:00
f6aca47604 xftp: implementation of XFTP client as web page (#1708)
* xftp: implementation of XFTP client as web page (rfc, low level functions)

* protocol, file descriptions, more cryptogrpahy, handshake encoding, etc.

* xftp server changes to support web slients: SNI-based certificate choice, CORS headers, OPTIONS request

* web handshake

* test for xftp web handshake

* xftp-web client functions, fix transmission encoding

* support description "redirect" in agent.ts and cross-platform compatibility tests (Haskell <> TypeScript)

* rfc: web transport

* client transport abstraction

* browser environment

* persistent client sessions

* move rfcs

* web page plan

* improve plan

* webpage implementation (not tested)

* fix test

* fix test 2

* fix test 3

* fixes and page test plan

* allow sending xftp client hello after handshake - for web clients that dont know if established connection exists

* page tests pass

* concurrent and padded hellos in the server

* update TS client to pad hellos

* fix tests

* preview:local

* local preview over https

* fixed https in the test page

* web test cert fixtures

* debug logging in web page and server

* remove debug logging in server/browser, run preview xftp server via cabal run to ensure the latest code is used

* debug logging for page sessions

* add plan

* improve error handling, handle browser reconnections/re-handshake

* fix

* debugging

* opfs fallback

* delete test screenshot

* xftp CLI to support link

* fix encoding for XFTPServerHandshake

* support redirect file descriptions in xftp CLI receive

* refactor CLI redirect

* xftp-web: fixes and multi-server upload (#1714)

* fix: await sodium.ready in crypto/keys.ts (+ digest.ts StateAddress cast)

* multi-server parallel upload, remove pickRandomServer

* fix worker message race: wait for ready signal before posting messages

* suppress vite build warnings: emptyOutDir, externals, chunkSizeWarningLimit

* fix Haskell web tests: use agent+server API, wrap server in array, suppress debug logs

* remove dead APIs: un-export connectXFTP, delete closeXFTP

* fix TypeScript errors in check:web (#1716)

- client.ts: cast globalThis.process to any for browser tsconfig,
  suppress node:http2 import, use any for Buffer/chunks, cast fetch body
- crypto.worker.ts: cast sha512_init() return to StateAddress

* fix: serialize worker message processing to prevent OPFS handle race

async onmessage allows interleaved execution at await points.
When downloadFileRaw fetches chunks from multiple servers in parallel,
concurrent handleDecryptAndStore calls both see downloadWriteHandle
as null and race on createSyncAccessHandle for the same file,
causing intermittent NoModificationAllowedError.

Chain message handlers on a promise queue so each runs to completion
before the next starts.

* xftp-web: prepare for npm publishing (#1715)

* prepare package.json for npm publishing

Remove private flag, add description/license/repository/publishConfig,
rename postinstall to pretest, add prepublishOnly, set files and main.

* stable output filenames in production build

* fix repository url format, expand files array

* embeddable component: scoped CSS, dark mode, i18n, events, share

- worker output to assets/ for single-directory deployment
- scoped all CSS under #app, removed global resets
- dark mode via .dark ancestor class
- progress ring reads colors from CSS custom properties
- i18n via window.__XFTP_I18N__ with t() helper
- configurable mount element via data-xftp-app attribute
- optional hashchange listener (data-no-hashchange)
- completion events: xftp:upload-complete, xftp:download-complete
- enhanced file-too-large error mentioning SimpleX app
- native share button via navigator.share

* deferred init and runtime server configuration

- data-defer-init attribute skips auto-initialization
- window.__XFTP_SERVERS__ overrides baked-in server list

* use relative base path for relocatable build output

* xftp-web: retry resets to default state, use innerHTML for errors

* xftp-web: only enter download mode for valid XFTP URIs in hash

* xftp-web: render UI before WASM is ready

Move sodium.ready await after UI initialization so the upload/download
interface appears instantly. WASM is only needed when user triggers
an actual upload or download. Dispatch xftp:ready event once WASM loads.

* xftp-web: CLS placeholder HTML and embedder CSS selectors

Add placeholder HTML to index.html so the page renders a styled card
before JS executes, preventing layout shift. Use a <template> element
with an inline script to swap to the download placeholder when the URL
hash indicates a file download. Auto-compute CSP SHA-256 hashes for
inline scripts in the vite build plugin.

Change all CSS selectors from #app to :is(#app, [data-xftp-app]) so
styles apply when the widget is embedded with data-xftp-app attribute.

* xftp-web: progress ring overhaul

Rewrite progress ring with smooth lerp animation, green checkmark on
completion, theme reactivity via MutationObserver, and per-phase color
variables (encrypt/upload/download/decrypt).

Show honest per-phase progress: each phase animates 0-100% independently
with a ring color change between phases. Add decrypt progress callback
from the web worker so the decryption phase tracks real chunk processing
instead of showing an indeterminate spinner.

Snap immediately on phase reset (0) and completion (1) to avoid
lingering partial progress. Clean up animation and observers via
destroy() in finally blocks.

* xftp-web: single progress ring for upload, simplify ring color

* xftp-web: single progress ring for download

* feat(xftp-web): granular progress for encrypt/decrypt phases

Add byte-level progress callbacks to encryptFile, decryptChunks,
and sha512Streaming by processing data in 256KB segments. Worker
reports fine-grained progress across all phases (encrypt+hash+write
for upload, read+hash+decrypt for download). Progress ring gains
fillTo method for smooth ease-out animation during minimum display
delays. Encrypt/decrypt phases fill their weighted regions (0-15%
and 85-99%) with real callbacks, with fillTo covering remaining
time when work finishes under the 1s minimum for files >= 100KB.

* rename package

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: shum <github.shum@liber.li>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-03-02 09:57:46 +00:00
Evgeny 6aadcf1f3f agent: lock rows for concurrent queries in PostgreSQL (#1688)
* agent: lock rows for concurrent queries in PostgreSQL

* fix race conditions in workers

* refactor
2026-01-08 11:09:58 +00:00
Evgeny 1329fc726f smp: support client notices (#1659)
* agent: support client notices

* improve

* fix, test

* rename

* cleanup

* send and process notices in more cases

* dont delete

* dont remove notice on other permanent errors

* dont remove notice if there is no notice ID in queue

* add server to error

* allow deleting

* only use notice if key hash matches
2025-10-17 18:34:59 +01:00
Evgeny cb3250e7b4 servers: better socket leak prevention during TLS handshake, add NetworkError type to better diagnose connection errors (#1619)
* servers: better socket leak prevention during TLS handshake

* log tcp connection errors

* more detailed network error

* log full address

* rename error

* add encodings for NetworkError

* refactor

* comment

* bind

* style

* remove parameters of NETWORK error from encoding
2025-09-02 16:07:37 +01:00
Evgeny beafac1f73 agent: make agent workers usable from other contexts (#1614) 2025-08-29 08:33:55 +01:00
Evgeny e48bedeaf2 servers: fix server pages when source code is not specified, include commit SHA in printed version and in web page. (#1608)
* smp server: fix server pages when source code is not specified

* servers: include git commit in version

* flexible alpn

* fix test

* fix ghc 8.10.7 build
2025-08-23 19:59:00 +01:00
Evgenyandsh f4e7469f96 xftp server: prometheus metrics (#1595)
* xftp server: prometheus metrics

* update

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-07-30 16:18:49 +01:00
Evgeny c0f357d817 servers: control port session improvements (#1591)
* servers: prohibit changing role during control port session

* quota for blocked queues

* allow disabling blocking and quota

* fix test

* fix INI file
2025-07-21 15:19:43 +01:00
Evgeny 36f05e272e agent: support different timeouts for interactive and background requests (#1582)
* agent: support different timeouts for interactive and background requests

* fix tests

* use one constructor for the first request and for retries
2025-07-07 09:38:52 +01:00
Evgeny da37384335 smp server: batch commands (#1559)
* protocol: refactor types and encoding

* clean

* smp server: batch commands (#1560)

* smp server: batch commands verification into one DB transaction

* ghc 8.10.7

* flatten transmission tuples

* diff

* only use batch logic if there is more than one transmission

* func

* reset NTF service when adding notifier

* version

* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"

This reverts commit 3df2425162.

* version

* Revert "version"

This reverts commit d80a6b74c5.
2025-06-12 23:05:04 +01:00
Evgeny 5241f5fe5e rfc: client certificates for servers using SMP protocol as clients (opertors' chat relays, notification servers, service bots) (#1534)
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)

* client certificates types (WIP)

* parameterize Transport

* protocol/schema/api changes

* agent API

* rename command

* agent subscriptions return local ClientServiceId to chat

* verify transmissions

* fix receiving client certificates, refactor

* ntf server: remove shared queue for all notification subscriptions (#1543)

* ntf server: remove shared queue for all notification subscriptions

* wait for subscriber with timeout

* safer

* refactor

* log

* remove unused

* WIP service subscriptions and associations, refactor

* process service subscriptions

* rename

* simplify switching subscriptions

* SMP service handshake with additional server handshake response

* notification delivery and STM persistence for services

* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error

* stats

* more stats

* rename SMP commands

* service subscriptions in ntf server agent (tests fail)

* fix

* refactor

* exports

* subscribe ntf server as service for associated queues

* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage

* INI option to support services

* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues

* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)

* fix test

* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription

* update RFC

* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands

* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission

* service errors, todos

* fix checkCredentials in ntf server, service errors

* refactor service auth

* refactor

* service agent: store returned queue count instead of expected

* refactor serverThread

* refactor serviceSig

* rename

* refactor, rename, test repeat NSUB service association

* respond with error to SUBS

* smp server: export/import service records between database and store log

* comment

* comments

* ghc 8.10.7
2025-06-06 08:03:47 +01:00
Evgeny 8e86c97a13 servers: include supported ALPNs in server transport config (#1557) 2025-06-03 13:35:35 +01:00
Evgeny 53b72469b6 refactor types (#1551) 2025-05-27 22:31:02 +01:00