* crypto: BBS scheme for anonymous credentials with multiple presentations
* verify
* add files to sources
* more files
* more files, use cabal 3.0
* fix path
* extensions
* switch libbbs to fork
* return either from keygen
* use only secret key to sign
* improve FFI
* simplify
* update libbbs to support iOS
* add commoncrypto flag
* bump libbbs
* reject input of wrong length
* ci: get submodules
---------
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
* agent: split SimplexNameDomain out of SimplexNameInfo
The type now separates the user-supplied type prefix (#/@) from the
domain itself:
data SimplexNameInfo = SimplexNameInfo
{ nameType :: SimplexNameType
, nameDomain :: SimplexNameDomain
}
data SimplexNameDomain = SimplexNameDomain
{ nameTLD :: SimplexTLD
, domain :: Text
, subDomain :: [Text]
}
The domain is independent of the contact-vs-public-group distinction —
the same dotted-labels structure applies to both. Future code that
needs to talk about a domain without committing to a name type (e.g.
server-side TLD-based registry lookup) can use SimplexNameDomain
directly.
fullDomainName now operates on SimplexNameDomain rather than the
full info wrapper. Parser, StrEncoding instance, and aeson derivations
updated accordingly. No external callers needed updating.
* agent: split StrEncoding instance for SimplexNameDomain
* agent: flatten TLD case + use unless guard
* agent: address review - strict domain parser, permissive channel
The per-(srvHost, provider) worker shards added in #1779 still funnel
all APNS sends through one HTTP2Client's reqQ, where a single process
thread calls sendRequest serially - one in-flight HTTP/2 stream at a
time, capping APNS throughput at 1/RTT.
sendRequestDirect bypasses the queue and invokes sendReq directly from
the calling worker, so concurrent workers open parallel HTTP/2 streams
on the shared APNS connection and the multiplexing happens on the wire.
* ntf-server: carry retry reason in PPRetryLater, log retries
Change PPRetryLater from nullary to PPRetryLater Text so the cause
(503 / 410-reason) propagates to the retry call site. Log a warning
at every retry attempt with provider, token id and reason.
* ntf-server: parallel push delivery via forkIO + per-srvHost lock
Fork delivery per notification, taking an MVar keyed by srvHost_ so
notifications from the same SMP server serialize while different
servers proceed concurrently. Switch APNS to sendRequestDirect so
concurrent deliveries share one HTTP/2 connection via stream
multiplexing rather than serializing through the client reqQ.
* ntf-server: single-flight push client creation via SessionVar
Match the take/create/wait pattern in Agent/Client.hs
(newProtocolClient / waitForProtocolClient). pushClients now wraps
clients in SessionVar (Either SomeException PushProviderClient) so
concurrent first-time access and concurrent retries collapse to a
single mkClient call; waiters observe the winner's result via
readTMVar (or its error). retryDeliver evicts the failing client by
SessionVar identity before re-fetching.
* use multiple queues and workers, remove semaphores and threads per notification
* fix
* retry connecting client
* fix
* move config
* fix
---------
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* xftp: add PostgreSQL backend design spec
* update doc
* adjust styling
* add implementation plan
* refactor: move usedStorage from FileStore to XFTPEnv
* refactor: add getUsedStorage, getFileCount, expiredFiles store functions
* refactor: change file store operations from STM to IO
* refactor: extract FileStoreClass typeclass, move STM impl to Store.STM
* refactor: make XFTPEnv and server polymorphic over FileStoreClass
* feat: add PostgreSQL store skeleton with schema migration
* feat: implement PostgresFileStore operations
* feat: add PostgreSQL INI config, store dispatch, startup validation
* feat: add database import/export CLI commands
* test: add PostgreSQL backend tests
* fix: map ForeignKeyViolation to AUTH in addRecipient
When a file is concurrently deleted while addRecipient runs, the FK
constraint on recipients.sender_id raises ForeignKeyViolation. Previously
this propagated as INTERNAL; now it returns AUTH (file not found).
* fix: only decrement usedStorage for uploaded files on expiration
expireServerFiles unconditionally subtracted file_size from usedStorage
for every expired file, including files that were never uploaded (no
file_path). Since reserve only increments usedStorage during upload,
expiring never-uploaded files caused usedStorage to drift negative.
* fix: handle setFilePath error in receiveServerFile
setFilePath result was discarded with void. If it failed (file deleted
concurrently, or double-upload where file_path IS NULL guard rejected
the second write), the server still reported FROk, incremented stats,
and left usedStorage permanently inflated. Now the error is checked:
on failure, reserved storage is released and AUTH is returned.
* fix: escape double quotes in COPY CSV status field
The status field (e.g. "blocked,reason=spam,notice={...}") is quoted in
CSV for COPY protocol, but embedded double quotes from BlockingInfo
notice (JSON) were not escaped. This could break CSV parsing during
import. Now double quotes are escaped as "" per CSV spec.
* fix: reject upload to blocked file in Postgres setFilePath
In Postgres mode, getFile returns a snapshot TVar for fileStatus. If a
file is blocked between getFile and setFilePath, the stale status check
passes but the upload should be rejected. Added status = 'active' to
the UPDATE WHERE clause so blocked files cannot receive uploads.
* fix: add CHECK constraint on file_size > 0
Prevents negative or zero file_size values at the database level.
Without this, corrupted data from import or direct DB access could
cause incorrect storage accounting (getUsedStorage sums file_size,
and expiredFiles casts to Word32 which wraps negative values).
* fix: check for existing data before database import
importFileStore now checks if the target database already contains
files and aborts with an error. Previously, importing into a non-empty
database would fail mid-COPY on duplicate primary keys, leaving the
database in a partially imported state.
* fix: clean up disk file when setFilePath fails in receiveServerFile
When setFilePath fails (file deleted or blocked concurrently, or
duplicate upload), the uploaded file was left orphaned on disk with
no DB record pointing to it. Now the file is removed on failure,
matching the cleanup in the receiveChunk error path.
* fix: check storeAction result in deleteOrBlockServerFile_
The store action result (deleteFile/blockFile) was discarded with void.
If the DB row was already deleted by a concurrent operation, the
function still decremented usedStorage, causing drift. Now the error
propagates via ExceptT, skipping the usedStorage adjustment.
* fix: check deleteFile result in expireServerFiles
deleteFile result was discarded with void. If a concurrent delete
already removed the file, deleteFile returned AUTH but usedStorage
was still decremented — causing double-decrement drift. Now the
usedStorage adjustment and filesExpired stat only run on success.
* refactor: merge STM store into Store.hs, parameterize server tests
- Move STMFileStore and its FileStoreClass instance from Store/STM.hs
back into Store.hs — the separate file was unnecessary indirection
for the always-present default implementation.
- Parameterize xftpFileTests over store backend using HSpec SpecWith
pattern (following SMP's serverTests approach). The same 11 tests
now run against both memory and PostgreSQL backends via a bracket
parameter, eliminating all *Pg test duplicates.
- Extract shared run* functions (runTestFileChunkDeliveryAddRecipients,
runTestWrongChunkSize, runTestFileChunkExpiration, runTestFileStorageQuota)
from inlined test bodies.
* refactor: clean up per good-code review
- Remove internal helpers from Postgres.hs export list (withDB, withDB',
handleDuplicate, assertUpdated, withLog are not imported externally)
- Replace local isNothing_ with Data.Maybe.isNothing in Env.hs
- Consolidate duplicate/unused imports in XFTPStoreTests.hs
- Add file_path IS NULL and status guards to STM setFilePath, matching
the Postgres implementation semantics
* test: parameterize XFTP server, agent and CLI tests over store backend
- xftpTest/xftpTest2/xftpTest4/xftpTestN now take XFTPTestBracket as
first argument, enabling the same test to run against both memory
and PostgreSQL backends.
- xftpFileTests (server tests), xftpAgentFileTests (agent tests), and
xftpCLIFileTests (CLI tests) are SpecWith-parameterized suites that
receive the bracket from HSpec's before combinator.
- Test.hs runs each parameterized suite twice: once with
xftpMemoryBracket, once with xftpPostgresBracket (CPP-guarded).
- STM-specific tests (store log restore/replay) stay in memory-only
xftpAgentTests. SNI/CORS tests stay in memory-only xftpServerTests.
* refactor: remove dead test wrappers after parameterization
Remove old non-parameterized test wrapper functions that were
superseded by the store-backend-parameterized test suites.
All test bodies (run* and _ functions) are preserved and called
from the parameterized specs. Clean up unused imports.
* feat: add manual tests and guide
* refactor: merge file_size CHECK into initial migration
* refactor: extract rowToFileRec shared by getFile sender/recipient paths
* refactor: parameterize XFTPServerConfig over store type
Embed XFTPStoreConfig s as serverStoreCfg field, matching SMP's
ServerConfig. runXFTPServer and newXFTPServerEnv now take a single
XFTPServerConfig s. Restore verifyCmd local helper structure.
* refactor: minimize diff in tests
Restore xftpServerTests and xftpAgentTests bodies to match master
byte-for-byte (only type signatures change for XFTPTestBracket
parameterization); inline the runTestXXX helpers that were split
on this branch.
* refactor: restore getFile position to match master
* refactor: rename withSTMFile back to withFile
* refactor: close store log inside closeFileStore for STM backend
Move STM store log close responsibility into closeFileStore to
match PostgresFileStore, removing the asymmetry where only PG's
close was self-contained.
STMFileStore holds the log in a TVar populated by newXFTPServerEnv
after readWriteFileStore; stopServer no longer needs the explicit
withFileLog closeStoreLog call. Writes still go through XFTPEnv.storeLog
via withFileLog (unchanged).
* refactor: rename XFTPTestBracket to XFTPTestServer
* fix: move file_size check from PG schema to store log import
* refactor: use SQL-standard type names in XFTP schema
* perf: batch expired file deletions with deleteFiles
* refactor: stream export instead of loading recipients into memory
* refactor: parameterize XFTP store with FSType singleton dispatch
* refactor: minimize diff per review feedback
* refactor: use types over strings, deduplicate parser
* refactor: always parse database store type, fail at startup
* fix compilation without postgresql
* refactor: always parse database store type, fail at startup
* lib: fix incorrect StrEncoding of Signature (it was not compatible with decoding, but was never used)
* align encoding with used in links (breaks backward compatibility)
* xftp-server: embed file download widget in XFTP server web page
When a URL has a hash fragment (>50 chars), the server page shows the
file download UI instead of the server info page. Embeds xftp-web
assets (JS, CSS, crypto worker) and protocol overlay with matching
website content. Overlay renders below the server navbar.
* xftp-server: fix overlay scroll lock, remove extra margin, fix dark SVG
* xftp-server: move file transfer widget to standalone /file page
* web: collapse all repeated Nothing sections in render
section_ only collapsed the first occurrence of a section when content
was Nothing, leaving subsequent sections with the same label intact.
This caused SMP server pages to show raw <x-xftpConfig> tags.
* xftp-server: update bundled css/js
* xftp-server: move file.html to xftp-server, rename xftp bundle dir
* web: remove unused server-info wrapper div
* refactor
* fix
---------
Co-authored-by: Evgeny <evgeny@poberezkin.com>
* web: parameterize generateSite, remove Embedded from library
Move embedFile/embedDir out of the library so it works when
simplexmq is consumed as a dependency. generateSite now accepts
mediaContent, wellKnown, and linkHtml as parameters.
* smp-server, xftp-server: embed static files in executables
Add shared apps/common/Embedded.hs with TH splices, update SMPWeb
and XFTPWeb to pass embedded content to generateSite, move
file-embed dependency from library to executables and test suite.
* refactor
* add export, move common files to Web subfolder
* fix .cabal
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* xftp-web: use XFTP server domain in share link, verify on download
Use a random XFTP server host from the file description as the link
origin instead of the current page domain. On download, verify
that the hosting domain matches one of the servers in the description.
* xftp-web: use first description server as link origin
* xftp-web: show wrong server error in download UI instead of blank page
* xftp: add web page for server information
* web: rename XFTP.Web to XFTPWeb, remove XFTP subdirectory
* refactor(xftp): remove storage quota from web page
* refactor
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* web: extract shared web module from smp-server
Move web serving infrastructure (warp, static files, HTML templating)
from apps/smp-server/web/Static.hs into library modules:
- Simplex.Messaging.Server.Web (generic web infra + templating)
- Simplex.Messaging.Server.Web.Embedded (TH-embedded assets)
Move static assets from apps/smp-server/static/ to
src/Simplex/Messaging/Server/Web/.
Move EmbeddedWebParams/WebHttpsParams from Server.Main to Server.Web.
Keep SMP-specific rendering (serverInformation) in apps/smp-server/SMP/Web.hs.
generateSite is now generic: takes pre-rendered HTML + link page paths,
enabling reuse by XFTP and NTF servers.
* web: add tests for templating engine
Tests for render, section_, item_, and timedTTLText functions
in Simplex.Messaging.Server.Web module.
* web: add serverInfoSubsts, serveStaticPageH2, safe port parsing
* web: rename SMP.Web to SMPWeb, remove SMP subdirectory
* fix(web): section_ collapsing sections with Just "" content
Commit e48bedea ("servers: fix server pages when source code is not
specified") changed section_ to treat Just "" the same as Nothing -
collapsing the section. The intent was to handle the sourceCode case
(empty string when not specified), but the guard
`not (B.null content)` also broke operator, admin, complaints, and
hosting - all of which legitimately use Just "" as a
section-present marker.
Before (correct):
Nothing -> before <> next
Just content -> before <> item_ label content inside <> ...
After (broken):
Just content | not (B.null content) -> ...
_ -> before <> next
Restore the original behavior: only Nothing collapses a section.
* refactor
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* xftp: implementation of XFTP client as web page (rfc, low level functions)
* protocol, file descriptions, more cryptogrpahy, handshake encoding, etc.
* xftp server changes to support web slients: SNI-based certificate choice, CORS headers, OPTIONS request
* web handshake
* test for xftp web handshake
* xftp-web client functions, fix transmission encoding
* support description "redirect" in agent.ts and cross-platform compatibility tests (Haskell <> TypeScript)
* rfc: web transport
* client transport abstraction
* browser environment
* persistent client sessions
* move rfcs
* web page plan
* improve plan
* webpage implementation (not tested)
* fix test
* fix test 2
* fix test 3
* fixes and page test plan
* allow sending xftp client hello after handshake - for web clients that dont know if established connection exists
* page tests pass
* concurrent and padded hellos in the server
* update TS client to pad hellos
* fix tests
* preview:local
* local preview over https
* fixed https in the test page
* web test cert fixtures
* debug logging in web page and server
* remove debug logging in server/browser, run preview xftp server via cabal run to ensure the latest code is used
* debug logging for page sessions
* add plan
* improve error handling, handle browser reconnections/re-handshake
* fix
* debugging
* opfs fallback
* delete test screenshot
* xftp CLI to support link
* fix encoding for XFTPServerHandshake
* support redirect file descriptions in xftp CLI receive
* refactor CLI redirect
* xftp-web: fixes and multi-server upload (#1714)
* fix: await sodium.ready in crypto/keys.ts (+ digest.ts StateAddress cast)
* multi-server parallel upload, remove pickRandomServer
* fix worker message race: wait for ready signal before posting messages
* suppress vite build warnings: emptyOutDir, externals, chunkSizeWarningLimit
* fix Haskell web tests: use agent+server API, wrap server in array, suppress debug logs
* remove dead APIs: un-export connectXFTP, delete closeXFTP
* fix TypeScript errors in check:web (#1716)
- client.ts: cast globalThis.process to any for browser tsconfig,
suppress node:http2 import, use any for Buffer/chunks, cast fetch body
- crypto.worker.ts: cast sha512_init() return to StateAddress
* fix: serialize worker message processing to prevent OPFS handle race
async onmessage allows interleaved execution at await points.
When downloadFileRaw fetches chunks from multiple servers in parallel,
concurrent handleDecryptAndStore calls both see downloadWriteHandle
as null and race on createSyncAccessHandle for the same file,
causing intermittent NoModificationAllowedError.
Chain message handlers on a promise queue so each runs to completion
before the next starts.
* xftp-web: prepare for npm publishing (#1715)
* prepare package.json for npm publishing
Remove private flag, add description/license/repository/publishConfig,
rename postinstall to pretest, add prepublishOnly, set files and main.
* stable output filenames in production build
* fix repository url format, expand files array
* embeddable component: scoped CSS, dark mode, i18n, events, share
- worker output to assets/ for single-directory deployment
- scoped all CSS under #app, removed global resets
- dark mode via .dark ancestor class
- progress ring reads colors from CSS custom properties
- i18n via window.__XFTP_I18N__ with t() helper
- configurable mount element via data-xftp-app attribute
- optional hashchange listener (data-no-hashchange)
- completion events: xftp:upload-complete, xftp:download-complete
- enhanced file-too-large error mentioning SimpleX app
- native share button via navigator.share
* deferred init and runtime server configuration
- data-defer-init attribute skips auto-initialization
- window.__XFTP_SERVERS__ overrides baked-in server list
* use relative base path for relocatable build output
* xftp-web: retry resets to default state, use innerHTML for errors
* xftp-web: only enter download mode for valid XFTP URIs in hash
* xftp-web: render UI before WASM is ready
Move sodium.ready await after UI initialization so the upload/download
interface appears instantly. WASM is only needed when user triggers
an actual upload or download. Dispatch xftp:ready event once WASM loads.
* xftp-web: CLS placeholder HTML and embedder CSS selectors
Add placeholder HTML to index.html so the page renders a styled card
before JS executes, preventing layout shift. Use a <template> element
with an inline script to swap to the download placeholder when the URL
hash indicates a file download. Auto-compute CSP SHA-256 hashes for
inline scripts in the vite build plugin.
Change all CSS selectors from #app to :is(#app, [data-xftp-app]) so
styles apply when the widget is embedded with data-xftp-app attribute.
* xftp-web: progress ring overhaul
Rewrite progress ring with smooth lerp animation, green checkmark on
completion, theme reactivity via MutationObserver, and per-phase color
variables (encrypt/upload/download/decrypt).
Show honest per-phase progress: each phase animates 0-100% independently
with a ring color change between phases. Add decrypt progress callback
from the web worker so the decryption phase tracks real chunk processing
instead of showing an indeterminate spinner.
Snap immediately on phase reset (0) and completion (1) to avoid
lingering partial progress. Clean up animation and observers via
destroy() in finally blocks.
* xftp-web: single progress ring for upload, simplify ring color
* xftp-web: single progress ring for download
* feat(xftp-web): granular progress for encrypt/decrypt phases
Add byte-level progress callbacks to encryptFile, decryptChunks,
and sha512Streaming by processing data in 256KB segments. Worker
reports fine-grained progress across all phases (encrypt+hash+write
for upload, read+hash+decrypt for download). Progress ring gains
fillTo method for smooth ease-out animation during minimum display
delays. Encrypt/decrypt phases fill their weighted regions (0-15%
and 85-99%) with real callbacks, with fillTo covering remaining
time when work finishes under the 1s minimum for files >= 100KB.
* rename package
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
---------
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: shum <github.shum@liber.li>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
* agent: use strict tables
* migrate existing tables to strict
* test: verify that all tables are strict
* fix column types for device_token and ntf_mode
* fix encodings and column types for ntf_sub_action and ntf_sub_smp_action
* update schema
* remove debug.trace
* log
* agent: make createConnection and setConnShortLink apis support setting all link data fields
* add functions
* refactor
* refactor
* fix tests
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* agent: support client notices
* improve
* fix, test
* rename
* cleanup
* send and process notices in more cases
* dont delete
* dont remove notice on other permanent errors
* dont remove notice if there is no notice ID in queue
* add server to error
* allow deleting
* only use notice if key hash matches
* agent: subscribe all connections
* query, version
* BoolInt
* add query to errors
* Revert "add query to errors"
This reverts commit 32a1f7fe11.
* fix optional field
* version
* limit number of in-flight subscriptions to 35000
* agent: optimize subscriptions memory usage more (do not store subscribed queues in memory) WIP
* use new session subscriptions data
* version
* remove old data structure
* remove version
* batch deletions
* test TSessionSubs
* comment
* smp server: limit by time the queues to export journal messages for
* pass queue/msg thresholds separately
* reset db connection on errors
* Revert "smp server: limit by time the queues to export journal messages for"
This reverts commit d3bc0cba4b.
* fix test compilation
* flag to expire messages
* improve test
* expire messages newer than quota
* smp server: store messages in PostgreSQL
* stored procedures to write and to expire messages
* function to export messages
* move all message functions to PostgreSQL, remove delete trigger
* comments
* import messages to db
* fix message import, add export
* fix export
* fix export
* fix compilation flags
* import messages line by line
* fix server start with database storage
* fix compilation
* comments
* smp server: fix server pages when source code is not specified
* servers: include git commit in version
* flexible alpn
* fix test
* fix ghc 8.10.7 build
* servers: prohibit changing role during control port session
* quota for blocked queues
* allow disabling blocking and quota
* fix test
* fix INI file
* smp protocol: create notification credentials via NEW command that creates the queue
* create ntf subscription for queues created with ntf credetials
* do not create ntf credentials when switching connection to another queue
* smp server: refactor subscriptions and delivery
* metric for time between MSG and ACK
* cleanup
* refactor pattern match for ghc 8.10.7
* time buckets
* split max time metric
* histogram
* fix
* agent: fix message delivery in case one of the connections has no snd queue for any reason - it could break delivery to all connections
* simplify
* comment
* refactor, also postpone failing on ratchet sync send prohibited errors
* postpone failing on connection errors to allow subsequent connections succeed
* don't exclude postgres modules for client_library flag
* fix build with client and server postgres flag
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* time buckets
* split max time metric
* histogram
* histogram for confirmed delivery times
* gaugehistogram
* fix created, _ in gauge_histogram
* remove comments
* fix metrics
* protocol: refactor types and encoding
* clean
* smp server: batch commands (#1560)
* smp server: batch commands verification into one DB transaction
* ghc 8.10.7
* flatten transmission tuples
* diff
* only use batch logic if there is more than one transmission
* func
* reset NTF service when adding notifier
* version
* Revert "smp server: use separate database pool for reading queues and creating service records (#1561)"
This reverts commit 3df2425162.
* version
* Revert "version"
This reverts commit d80a6b74c5.
* agent: use PQ keys in contact request data inside link container (but not in contact request link); use PQ keys in invitations sent to contact addresses
* do not use PQ keys in the link with old address versions
* log alpn
* always use HTTP when SNI is sent, regardless of ALPN
* decide credential based on SNI
* do not use web port in SMP/Ntf servers connecting to SMP servers
* simpler
* refactor
* fix
* rfc: client certificates for high volume clients (opertors' chat relays, notification servers, service bots)
* client certificates types (WIP)
* parameterize Transport
* protocol/schema/api changes
* agent API
* rename command
* agent subscriptions return local ClientServiceId to chat
* verify transmissions
* fix receiving client certificates, refactor
* ntf server: remove shared queue for all notification subscriptions (#1543)
* ntf server: remove shared queue for all notification subscriptions
* wait for subscriber with timeout
* safer
* refactor
* log
* remove unused
* WIP service subscriptions and associations, refactor
* process service subscriptions
* rename
* simplify switching subscriptions
* SMP service handshake with additional server handshake response
* notification delivery and STM persistence for services
* smp server: database storage, store log, fix encoding for STORE error, replace String with Text in locks and error
* stats
* more stats
* rename SMP commands
* service subscriptions in ntf server agent (tests fail)
* fix
* refactor
* exports
* subscribe ntf server as service for associated queues
* test ntf service connection, fix SOKS response, fix service associations not removed in STM storage
* INI option to support services
* ntf server: downgrade subscriptions when service is no longer supported, track counts of subscribed queues
* smp protocol: include service certificate fingerprint in the string signed over with entity key (TODO two tests fail)
* fix test
* ntf server prometheus stats, use Int64 in SOKS/ENDS responses (to avoid conversions), additional error status for ntf subscription
* update RFC
* refactor useServiceAuth to avoid ad hoc decisions about which commands use service signatures, and to prohibit service signatures on other commands
* remove duplicate service signature syntax check from checkCredentials, it is checked in verifyTransmission
* service errors, todos
* fix checkCredentials in ntf server, service errors
* refactor service auth
* refactor
* service agent: store returned queue count instead of expected
* refactor serverThread
* refactor serviceSig
* rename
* refactor, rename, test repeat NSUB service association
* respond with error to SUBS
* smp server: export/import service records between database and store log
* comment
* comments
* ghc 8.10.7
* agent: setInvitationShortLink api
* Eq instance
* allow changing link data on server, refactor
* fix
* encodings
* remove link data after connection
* Revert "encodings"
This reverts commit f8e254cca9.
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* parameterize transport by peer type (client/server)
* LogDebug level when test is retried
* support "flipped" HTTP2, fix test retry to avoid retrying pending tests
* move sync to the end of the tests
* agent: return error and message absence differently when getting notification messages
* fix test
* mapM
* inline nse functions, release lock on error or no message
* agent: handle cases when last message ts is not set for notifications; set last ts for "stale" notifications when messages expired and queue is empty, to prevent repeated processing
* only log errors if they exist
* only set last ts for queue that delivered notification
* ntf server, agent: send all periodic notifications from one thread, only to old active clients or new clients with periodic notification mode
* send different type via subscription queues
* option to compact store log on start
* ntf server: skip duplicates when importing tokens and subscriptions
* skip imported last notifications when no token or subscription present
* fix skipped imported notifications count
* all tests
* fix test
* ntf server: allow retries when creating subscriptions, prohibit subscriptions with the same queue but another notifier key or token
* sync files in the test
* refactor
* agent: option to use web port by default for preset servers only
* shorten/restore short links in agent, add encodings for SMP web port setting
* decouple preset domains from preset servers for short links
* refactor, rename
* smp server: short links and owners for channels
* types
* support mutliple rcv keys
* fix down migration, test/create server schema dump
* reduce schema dump
* parameterize type for link data by connection type
* return full connection link data
* test version
* change short link encoding
* test: print pg_dump output
* server pages, link encoding
* fix connection request when queue data and sender ID are created for old servers
* test, change pattern
* ci: install postgresql tools in runner (#1507)
* ci: install postgresql tools in runner
* ci: docker shell abort on error
* fix pattern for ghc 8.10.7
* patch ConnReqUriData SMP encoding to preserve queue mode after decoding
* test for RKEY
* fix/test store log with RKEY
---------
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
* agent: padded encryption for link data, tests
* lambda
* test short links via proxy
* tests: server persistence with short links
* rfc: group links
* shorten, restore, test short links encoding
* rfc
* agent: join connection when 1-time invitation short link is already secured
* do not pass short link to join
* delete short link record after connection
* smp server: remove locks for deleted queues, additional statistics for objects in memory
* version
* reduce queue cache usage
* less caching, refactor
* comments
* revert version
* smp protocol: short links types and other changes from RFC
* add fields for queue link ID and data
* create queue and ntf credentials with NEW command
* all tests
* simplfiy types, update rfc
* update rfc
* include SenderId in NEW request in case queue data is sent
* store queue data and generate link ID if needed
* update rfc
* agent API and types
* SMP commands and persistence for short links
* SMP client functions for short links
* agent client functions for short links
* create rcv queue with short link (TODO secret_box)
* encryption and encoding for link data, postgres client migration
* test creating short link
* get link and data, tests
* comments
* type signature
* smp server: use COPY to import store log to postgres db
* compact queues when importing to postgres
* mempty
* version
* handle errors while expiring, mask async exceptions while getting queue
* whitespace
* version
* smp server: split postgres support to a separate executable, to not require postgres library in the main binary
* comments
* enable server_postgres flag by default, add CPP option to test
* refactor
* change default for server_postgres to False
* diff
* smp server: expire only active queues
* version
* do not cache all queues while processing expirations
* refactor
* foldWithOptions_
* version
* use shared lock when expiring all queues
* use TMVar
* comment
* rename
* remove fold options
* do not create locks in the Map for temporarily loaded queues
* fix
* revert version
* smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration)
* refactor
* remove comment
* smp server: queue store typeclass
* parameterize JournalMsgStore
* typeclass for queue store
* postgres WIP
* compiles, passes tests
* remove StoreType
* split migrations
* progress
* addQueueRec
* reduce type spaghetti
* remove addQueue from typeclass definition
* getQueue
* test postgres storage in SMP server
* fix schema
* comment
* import queues to postgresql
* import queues to postgresql
* log
* fix test
* counts
* ci: test smp server with postgres backend (#1463)
* ci: test smp server with postgres backend
* postgres service
* attempt
* attempt
* empty
* empty
* PGHOST attempt
* PGHOST + softlink attempt
* only softlink attempt
* working attempt (PGHOST)
* remove env var
* empty
* do not start server without DB schema, do not import when schema exists
* export database
* enable all tests, disable two tests
* option for migration confirmation
* comments
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* smp server: remove empty journals when opening message queue
* update, do not backup state
* test
* version
* do not close queue state when queue is opened for writing
* comment
* quota = 4
* refactor openMsgQueue to prevent extra state backups
* use interval in config
* version, expire backups after 5 min
* refactor
* test
* agent: store shared message body only once (when it is the same across messages when batching)
* rename
* refactor
* refactor
* save bodies and messages in single transaction
* comment
* comment
* comment
* box
* mapME
* box
* ValueOrRef
* remove instances
* refactor
* comments
* test
* refactor
* mapAccumLM compatibility with ghc 8.10.7
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* agent: save message body once (plan, schema)
* split
* new type
* bs
* encrypt on delivery
* schema
* fix test
* check pad size
* rename
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* ntf server: record token invalidation reason, add date of the last token activity
* update time
* rename
* optional
* include token ID in delivery error
* version
* protocol version
* fix, log error
* xftp server: use recipient ID in control port to delete and block files
* cap smp proxy agent version at 10
* version
* fix prometheus
* fix
* remove old version support
* log connection parameter on error
* tests
* log sent command tag
* log error and client version
* cap proxy version for previous destination server
* comment, test
* remove logging tag
* remove logs
* version
* SMP version 14
* version
* remove comments
* version
* servers: blocking records for content moderation
* update
* encode BLOCKED as AUTH in old versions
* update
* unblock queue command
* test, status command
* server: support server roles and operators
* make server operator optional
* allRoles
* fix test
* different server host in tests
* remove ServerCfg fields used only in UI
* comments
* choose different server for invitation when connecting via address
* fix test in ghc8107
* simplify
* smp server: do not open/read journal message queues that are known to be empty
* cleanup
* version
* close empty queues on first subscription
* revert version
* smp server: combine queue and message store into one class (WIP)
* keep deleted queue tombstones to prevent race conditions and errors when restoring
* move store log from server to store implementations
* STMQueueStore type class
* fix store closed when messages expired, handle store writing errors
* types
* version
* fix recovery from missing write journal, tests
* version
* ntf: get messages for multiple last notifications (#1352)
* ntf: separate get ntf conns api (#1379)
* ntf: separate get ntf conns api
* nonempty
* update
* update
* remove single get api
* fix test
* refactor
* refactor
* ntf: batch get connections (#1387)
* ntf: batch get apis
* works
* fix
* fix
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* smp server: fixed logging format for journal store errors
* version
* colon
* logs
* refactor
* space
* remove comment
* log file name in fixFileSize
* logError
* stricter
* process all queues more efficiently
* use monoid for queue processing
* expire messages concurrently
* concurrently 2
* Revert "concurrently 2"
This reverts commit c1aee1f22c.
* Revert "expire messages concurrently"
This reverts commit fc53137cdb.
* show queue directory or ID in errors
* foldM
* mask_
* try
* mask more
* refactor
* command to delete journal
* uninterruptibleMask_ when writing to state file
* fix ghc8.10.7
* version
* revert version
* smp server: journal store queue state validation
* validate state on update, refactor
* handle message parsing errors
* typo
* fix test, throw exception when file is smaller than expected
* fix
* fix test in ghc 8.10.7
* core: updated journal store API
* parameterize JournalState by read/write type
* collect stats when importing/expiring
* compare stored stats and store for messages and notifications
* simplify
* smp server: remove STM function from MsgStore
* polymorphic MsgStore
* jourmal storage for messages (WIP)
* more journal, test setup
* writeMsg
* test
* tryDelMsg
* delMsgQueue
* remove MsgStoreClass instance of existential wrapper for Msg stores
* store config
* extract common logic out of store instances
* add store type to config
* open journals, cache last message, tests pass
* CLI commands
* refactor import/export messages
* cli commands to import/export journal message store
* export journal without draining, import/export tests
* journal command
* import/export progress
* better progress info
* only log queue state once when importing
* logs
* handle IO errors in journal store, return as STORE error
* recover from state file errors
* fix message files after crash
* fix messages folder
* .401
* stats for undelivered notifications
* logs, stats
* control port show ntf client IDs
* check that Ntf client is still current and that queue is not full, drop notifications otherwise
* prevent losing notifications when client is not current or queue full
* add log when no notifications, remove some logs
* reduce STM transaction
* revert version change
* smp server: pass server information via CLI during server initialization
* more info
* enable client expiration by default, disable port 8000
* update
* ntf server: control port
* version .405
* control
* use own_server_domains from INI file
* fix subs by server in control port
* bigger queues
* ntf server: only print subscriptions per own server when they are > 0
* fix tests
* revert version change
* dont import listThreads in ghc 8.10.7
* smp-server: Allow serving HTTPS and transport on the same port
* update rfc
* servers: refactor TLS credentials
* provide server credentials in SNI hook
* determine TLS server params dynamically, when starting the server
* remove alpn from TransportServerConfig to decide it dynamically where server is started
* diff
* combine HTTP and SMP on the shared port
* Update to SockAddr
* Fix params and web.https parser
* Switch fork urls
* WIP: add smpServerTestStatic test
* Update warp-tls repo
* shared connection tests
* cleanup
* Add protocol tests
* rename cert file, enable both ports and web by default
* terminate with message on missing credentials
* test cert file
* client option to use port 443 as default SMP port
* use SNI in non-SMP clients
* supported
* remove TODO
* advice
* fix test build
* Add RSA-4096 check for web creds, fix test
* Remove directory listing from static app
* message
* messages
* update log tests
---------
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* servers: refactor TLS credentials
* provide server credentials in SNI hook
* determine TLS server params dynamically, when starting the server
* remove alpn from TransportServerConfig to decide it dynamically where server is started
* agent: transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes
* fix test
* agent: support socks proxy without isolate-by-auth, with and without credentials
* add unit tests
* make xftp use correct SOCKS credentials
* rename
* support ipv6 in brackets, test parsing
* constant
* textToHostMode
* space
* agent: don't prohibit deletion of notifications token if different is passed (fixes turning notifications off after importing database)
* ignore error
* notify
* smp protocol: send DELD when subscribed queue is deleted
* fix, test
* refactor
* send DELD event only if the client supports it (version 10); send END otherwise
* fix test
* notify on notifier rotation
* increase test delays
* smp server: add created/updated/used date to queues to manage expiration, all: make Map updates strict in value
* remove strict
* remove time precision
* diff
* style
* only update when time changed
* smp server: fewer map updates on re-subscriptions
* temp version
* replace Client with ClientId in queues
* version
* version
* comments
* reduce threads when sending ENDs
* revert version
* put DRG state to IORef, split STM transaction of sending notification (#1288)
* put DRG state to IORef, split STM transaction of sending notification
* remove comment
* remove comment
* add comment
* revert version
* newtype for server entity IDs, fix TRcvQueues
* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"
This reverts commit 517933d189.
* logServer
* put DRG state to IORef, split STM transaction of sending notification (#1288)
* put DRG state to IORef, split STM transaction of sending notification
* remove comment
* remove comment
* add comment
* revert version
* smp server: get message queue faster, avoiding STM contention if queue exists
* IORef for counter
* Revert "put DRG state to IORef, split STM transaction of sending notification (#1288)"
This reverts commit 517933d189.
* version
* remove IORef
* split notification delivery transations
* revert version
* agent: support new connection user id update
* another way for assertion
* add more tests to setConnUserId
* remove fdescribes
* allow rcv connection to change user id
* add functional test to api
* remove fdescribe
* refactor
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* agent: use weak ThreadId and forkIO in workers instead of async (reduce memory)
* agent: do not start and exit delivery workers when there are no messages to deliver (#1264)
* agent: exit delivery workers when no messages to deliver
* only start delivery workers when there are pending messages
* fix
* focus test
* enable all tests
* lift
* do not exit workers when there is no work
* agent: retry loop that resumes subscriptions as soon as agent is moved to foreground, suspend retry loops while agent is suspended
* reset retry enterval when moving to foreground
* account for network state too
* simplify
* typo
* simplify
* ntf server: only mark subscriptions as pending if the disconnected client is current
* add sessionId to subscribed queue
* add sessionId to subscriptions in ntf server agent
* fix
* agent: fix possible dead lock between sending and receiving messages, stress test for message delivery
* deliver events after the lock is released
* delayed delivery in command processing too
* tests: increase message expiration time
* rfc: faster handshake protocol
* update
* 1 message
* SKEY
* use SKEY for both parties
* test
* update doc
* NEW command parameter
* add k=s param to queue URI
* fix
* add sndSecure field to queues
* make sender key non-optional in SndQueue (WIP, tests fail)
* fast handshake sometimes works (many tests fail)
* correctly handle SKEY retries, avoiding to re-generate the keys
* handle SKEY retries during async connection
* fix most tests (1 test fails)
* remove do
* fix contact requests encoding/tests
* export
* fix: ignore duplicate confirmations, fixes testBatchedPendingMessages
* do not store sndSecure in store log if it is false to allow server downgrade
* add connection invitation encoding tests
* agent: report correct errors from xftp handshake so they are treated as temporary
* disable slow servers test
* remove comments
* all tests
* remove duplicate functions
* agent: report correct errors from xftp handshake so they are treated as temporary
* disable slow servers test
* remove comments
* all tests
* remove duplicate functions
* SMP server information
* fix tests
* country codes
* smp-server: serve contact and link pages from static files (#1084)
* smp-server: serve contact and link pages from static files
* generate index
* use params from ini
* render using ServerInformation
* tweak templates
* update
* fix some html
* smp-server: fix layout (#1097)
* smp-server: fix layout
* port fixes to link page
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* update server information page
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>
* update server info
* web: improve server info page design (#1166)
* web: improve server info page design
* web: fix font errors & some tags
* web: improve contact & invitation page layout and header
* update
* remove unused files/css
* cleanup
* fix link page
* remove unused font links
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* show contact address as is
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: M. Sarmad Qadeer <MSarmadQadeer@gmail.com>
* smp: put client proxy command processing threads under a shared semaphore
* add LIMITS.max_proc_threads to server config
* rename to PROXY.client_concurrency
* retry on strictly greater than max concurrency
* set default to 16
* rename
* fix limit
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* agent: aggregate multiple expired subscription responses into a single UP event
* clean up
* refactor processing of expired responses
* refactor
* refactor 2
* refactor unexpectedResponse
* smp: use session vars for reconnecting small agent
* process errors
* split session and protocol functions
* add active flag to agent
* actually invoke agent shutdown
* close proxy agent too
* restore stopping ntf subscribers
* agent: do not mark subscriptions on expired sessions as active, do mark delayed subscriptions as active on the same session, SUBOK response in the next SMP protocol version
* client: prevent sub actions from zombie sessions (#1122)
* client: prevent sub actions from zombie sessions
* error handling
* add AERR to pass background errors to client
* switch to activeClientSession
* put closeClient under activeClientSession
* rename
* remove AERR, do not skip processing
* move check and state update to one transaction
* catch extra UPs
* fix
* check queue is still pending before making it active
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* do not forward agent error
* revert not expiring sending subs
* fixes
* track subscription responses better
* add pending connection
* Revert "revert not expiring sending subs"
This reverts commit 4310a69391.
* do not expire sending commands
* rename
* fix race
* function
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* proxy: negotiate client-relay version, include it in PFWD commands and in encrypted forwarded transmissions
* rename
* inline
* comment
* use correct server version when encoding forwarded commands
* proxy: include delivery path in SENT event
* send MWARN event to user on server version or host more errors
* Revert "proxy: include delivery path in SENT event"
This reverts commit 5c476718ec.
* transport: reduce ping traffic
* make pings opt-in, enable automatically with SUB commands
* fix reduced delays
* enable pings on MSG too
* rename pingErrorCount
* check timeout counter even when not sending pings
* clean up
* reset timeout error count on any event
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* SMP proxy: low level client and server implementation
* SMP proxy: server implementation (#1098)
* wip
* PRXY command
* progress
* SMP Proxy: client-level implementation (#1101)
* buildable
* encode messages
* update pkey
* fix queue types
* wrap SEND in proxy lookup
* WIP proxy client
* WIP
* post-rebase fixes
* encode something with something
* cleanup
* update
* fix nonce/corrId in batchingTests
* WIP: dig into createSMPProxySession
* agent
* test progress
* pass the test
* parameterize transport handle with transport peer to include server certificate (#1100)
* parameterize transport handle with transport peer to include server certificate
* include server certificate into THandle
* load server chain and sign key
* fix key type
* fix for 8.10
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* cleanup
* add 2-server test
* remove subsumed test
* checkCredentials for BrokerMsg
* skip batching tests
* remove userId param
* remove agent changes
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* remove unused type
* icrease test timeout
* reduce transport block
* envelope sizes
* don't fork unless have proxied commands to process
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* parameterize transport handle with transport peer to include server certificate
* include server certificate into THandle
* load server chain and sign key
* fix key type
* fix for 8.10
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* parameterize transport handle with transport peer to include server certificate
* include server certificate into THandle
* load server chain and sign key
* fix key type
* fix for 8.10
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* test with failing files (in progress)
* print
* add replica uploading state
* Revert "add replica uploading state"
This reverts commit 7068213aa6.
* <=
* fix
* prints
* test no redundancy
* all tests no redundancy
* revert delay
* refactor
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* log control port commands
* auth
* add auth to xftp, config and commands
* log missing auth
* put smp save under auth
* corrections
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* convert compress to a wrapper with passthrough fallback
* add compress1 for non-batched compression
* use original size as upper bound for scratch
* refactor
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* envelope sizes dependent on PQ encryption (WIP)
* add "supported" flag to ratchets, update this flag on ratchet resync
* change connection PQ status on sendMessage
* comment, fix
* refactor
* fix agent performance leak when re-connecting clients, optimize SMP server
* let the clients remove themselves from their clientvars
* fix another test
* do not call removeSubs when AgentClient is not active
* revert some changes
* revert more, refactor
* comment
* rename
* refactor
* refactor
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* smp: command authorization
* fix encoding, most tests
* remove old tests
* authorize via crypto_box
* extract authenticator to Crypto module
* make TransmissionAuth Maybe
* rfc
* support authenticators in NTF protocol, test matrix (no backwards compatibility yet from new clients to old servers)
* fix/add tests, add version config to "small" agent
* separate client and server versions for SMP protocol
* test batching SMP v7
* do not send session ID in each transmission
* refactor auth verification in the server, split tests
* server "warm up" fixes timing test
* uncomment SUB timing test
* comments, disable two timing tests
* rename version
* increase auth timing test failure threshold
* use different algorithms to authorize snd/rcv commands, use random correlation ID
* transport: fetch and store server certificate (#985)
* THandleParams (WIP, does not compile)
* transport: fetch and store server certificate
* smp: add getOnlinePubKey example to smpClientHandshake
* add server certs and sign authPub
* cleanup
* update
* style
* load server certs from test fixtures
* sign ntf authPubKey
* fix onServerCertificate
* increase delay before sending messages
* require certificate with key in SMP server handshake
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* remove dhSecret from THandle
* remove v8, merge all changes to one version
* parameterize THandle
* rfc: transmission ecnryption
* Revert "parameterize THandle"
This reverts commit 75adfc94fb.
* use batch syntax for ntf server commands
* separate encodeTransmission when there is no key
* typo
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* rename
* diff
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* xftp: add URI encoding for FileDescription
* tweak URI
* allow smaller blocks
* draft xftpReceiveFileFollow' and xftpSendFilePublic'
* add sending with redirect
* allow 64k chunks
* add migrations with redirect fields
* add test case
* fix deadlock
* revert CLI code
* WIP: working send/receive via URI
* fix field ambiguity
* cleanup
* update agent db schema
* update minimal chunk size
* add rfc
* apply suggestions from code review
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* add createRcvFileRedirect
* extract Simplex.Messaging.ServiceScheme and reuse for files
* update db schema
* check size/digest on receive complete
* cleanup
* use SIZE/DIGEST errors for redirects too
* split digest/size errors from redirect checks
* fix redirect error encoding
* rename RedirectMeta to RedirectFileInfo
* use query encoding for file URI
* group maybe fields under RcvFileRedirect
* add extras field
* update rfc
* add extras encoding and no-redirect tests
* fix toStrict for old ghc
* extra client data in file descr URI
* remove decoded yaml file
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* smp-server: add gen-online CLI command
* use CN and algo from old certificate
* add cert checks to test
* rename command
* fix test
* cert
---------
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
* tests: add ntf case with multiple Ntf servers
* simplify test
* fix for master
* add server switch test
* add server switch test
* add message test for ntf server switch
* smp-server: check queue balance in stats vs store
* smp-server: add msgExpired stats
* add msgExpired stats
* split expire/stats transactions
* count and pass msgExpired explicitly
* save/load qCount and use it for checking store
* xftp: test file reception - shouldn't get stuck if file is deleted on server
* comment
* expiration test
* approach
* wip
* sort by retries in other works, revert some diff
* revert diff
* modify tests
* refactor
* refactor
* remove prints
* apply to other workers
* remove import
* comment
* refactor
* revert queue size
* fix test
* rename
* comment, correct number of retries
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* control: add delete command
* logDeleteQueue only when found
* use default StrEncoding for CPDelete arg
* move stats update from main transaction
* use size
* stabilize AUTH timing tests
* more iterations
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* agent: mark work items failed (WIP)
* add tests, created_at
* getWorkItem for snd and rcv files
* fix
* tests
* fix
* tests
* test
* tests
* rename
* fix,refactor
* add indexes
* update schema
* do not try to get more work when resuming an existing worker
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* agent: notify about polled message processing (for iOS notifications)
* optionally keep key and support re-opening database
* exports
* test that cannot reopen when created with keepKey: false
* set max number of messages to receive for a notification to 3
* Revert "raise mtl version to 2.3.1 (#912)"
This reverts commit 117168ccce.
* Revert "expand dependency ranges (#911)"
This reverts commit 90a8fc91d3.
* fix test
* remove unnecessary forks
* make text version dependent on GHC version
* text
* remove stack.yaml
* bytestring
* bytestring
* hpack 0.35
* mtl
* add sntrup761 source
* it compiles
* Wrap bindings in non-FFI types
Test passes with a dummy RNG.
* pass ChaChaDRG via FunPtr
* Add iOS smoke test at createAgentStore
* style
* add "ssl" library dep
Attempt to fix missing _SHA512 symbol on macos.
* remove sha512 wrapper and use openssl directly
* restore names, remove dummy RNG
* Revert "remove sha512 wrapper and use openssl directly"
This reverts commit f9f7781f09.
* restore code from RFC
* shorter names
* enable all tests
* remove run test
---------
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
* Cut transport server to allow custom tcp servers
Allows socket inspection before wrapping up in a transport/prototocol.
* rename
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* Add X509 cert and TLS credentials generator
* Expand Crypto toolkit and rewrite tls credentials with it
* Exclude X keys from SignatureAlgorithmX509 and TLS.PrivKey
* Add helpers for DB marshalling and fingerprints
* Derive public key from private
* remove module name from selectors
* Remove StrEncoding (PrivateKey Ed25519)
* remove comment
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* Split HTTP2 server and client setup
For attaching to an existing TLS session.
* Add genTlsCredentials
* Allow chain construction from separate credentials
So the CA may be stored and leaf ephemeral.
* Rewrap X509 fingerprint into simplex KeyHash used in transport
* Fix docstring
* Remove TLS.Credentials generator
* Trace auto-subs flag
* Replace Bools with SubscriptionMode
* Handle SMOnlyCreate
* Wire remaining todos
* Update tests and fix
* Bump protocol level
* Apply suggestions from code review
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* Scrub needs_sub from agent DB
* Scrub a few more needSubs from the agent api
* change API, fix test
* agent: do not subscribe to queue when creating reply queue
* fix encoding
* WIP: SMOnlyCreate test
* Add SM guard for confirmQueue
Allows the test case to pump the allowConnection
reply without getting PROHIBITED.
* Remove tracing
* add noMessages, remove unnecessary getConnectionMessage from test
* add sending messages to the test
---------
Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* server: add control port commands for clients and ghc threads (#835)
* Add stats-rts control query
With supporting ghc-options that would provide the data.
* Add CPSkip command
Allows spamming empty lines a few times to clean up the view.
* server: Add CP commands to enumerate clients and threads
* style
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* use base64 encoding for session ID
* fromMaybe
* whitespace
---------
Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
* Add env lookups for smp server paths
Allows running smp-server without touching system paths.
Can be helpful for running multiple instances.
* allow empty env var values
* diff
* fix
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* agent: catch IO errors correctly in MonadError
* correction
* correction
* utils
* agentFinally to catch IO exceptions in ExceptT
* rename
* remove, inline
* rename utils
* utils unit test
* test to show catch and finally problems
* tryAllErrors
* enable all tests
* make timeouts for batched functions dependent on the number of batches
* fix
* refactor
* refactor
* change import
* refactor
* rename function
* rename
* refactor
* remove space
* ntf server: better batching and logging
* reduce batch delay for ntf server
* comments
* 5.1.3, ntf 1.4.2
* more logging
* more logging
* split large batches, more logging
* remove some logs
* docker entrypoint: DRY store log backup
Don't Repeat Yourself
Set a variable for the full path to the source file.
Create the backup path by appending an extension from `date` output.
Also fixed quoting and switched to an `if` block.
Clean up block level variables.
* docker entrypoint: use ISO 8601 format
The previous format discards information about the local time zone.
* docker entrypoint: always use UTC
Now the format always ends in +00:00, and we can ignore that part again.
* docker entrypoint: provide the date format again
* docker entrypoint: remove time zone from the date format
* docker entrypoint: use an unambiguous date format
Present a leading zero before the month:
YYYY-0MM-DD
Both YYYY-MM-DD and YYYY-DD-MM are used by people and can be confusing in the beginning of the month.
* docker entrypoint: use appropriate quoting
Without avoiding field splitting a password containing a space changes the number of arguments being set.
* docker entrypoint: use explicit braces for custom variables
Make the intentions clear and don't assume the user knows the special cases for when variables won't be extended.
Example:
word=animal
words=mistake
echo "$words vs $word vs ${word}s"
* docker entrypoint-xftp-server: braces and quoting
* docker entrypoint-xftp-server: backup block
* docker entrypoints: explain date format in a comment
* switched from long to short option to date for POSIX
* docker entrypoint-smp-server: explain date format
* docker entrypoint-xftp-server: explain date format
* docker entrypoints: further explain format
I fixed the case to match the date format letters.
Also, use words to explain since I don't want everyone to need to read about date formats to understand.
* docker entrypoints: only quote letters
I was either going to quote the dashes too or stop quoting the colons.
Having less quotes was more readable.
* Revert "docker entrypoint: use an unambiguous date format"
This reverts commit ba2a93bad9.
* docker entrypoints: remove %F
* docker entrypoint-smp-server: remove %F
Used part of the explicit ISO 8601 format,
provided by the coreutils date invocation guide.
* docker entrypoint-xftp-server: remove %F
Used part of the explicit ISO 8601 format,
provided by the coreutils date invocation guide.
* Add GH Actions workflow to build and push Docker image on release
* Use a matrix to support building other applications
* Rename application from simplexmq to smp-server
* Remove linux/386 platform
* Remove linux/arm64 platform
* xftp: agent API to set and test servers
* ProtocolTestStep
* update agent API for XFTP servers
* ci: update ubuntu versions
* disable test hanging on ubuntu
* ci: disable 2 tests on linux only, switch to ubuntu 20 and 22
* fix platform name
* keep ubuntu 22 binaries
* Revert "keep ubuntu 22 binaries"
This reverts commit a1bbb12870.
* skip 1 more test
* skip 1 more test
* log os
* log os
* unconditionally skip test
* skip 1 more test in CI
* fix tests
* add llvm to build stage
running docker build on Apple Sillicon would result in this error:
#14 49.39 <no location info>: error:
#14 49.39 Warning: Couldn't figure out LLVM version!
#14 49.39 Make sure you have installed LLVM between [9 and 13)
I assume it can’t find llvm/clang?
* install using unattended helper script
the current haskell is hardcoded to x86, so switch to using haskell’s official helper script which supports arm
also need to bump up setting the path, since ghcup is no longer in /usr/bin
reference: https://www.haskell.org/ghcup/guide/#continuous-integration
* add numa headers/libs
docker build (specifically haskell install step) would fail with :
#7 107.2 utils/ghc-cabal/dist-install/build/tmp/ghc-cabal: error while loading shared libraries: libnuma.so.1: cannot open shared object file: No such file or directory
I think this is because numa isn’t installed on arm instances of focal by default, but it is in later versions
anyway this fixes that
* also add numa to final
The image builds at this point! But it fails to boot on arm - again with a numa error.
This fixes that.
* don’t specify llvm version
---------
Co-authored-by: +shyfire131 <shyfire131@shyfire131.net>
- chunk download doesn't loop on permanent errors
- decryption errors are considered permanent - local worker doesn't retry
- update replica retries; to do - consider use for this field, or remove it
- rcv file Error status - to prevent repeat reads of chunks for download, files for decryption; also plan to use it for filtering on cleanup
- error string saved in separate field for debugging (not part of status type)
- agent event for rcv file errors
Noticed that the environment variables for SMP server when using Docker were improperly quoted, so I corrected them.
This will fix an issue with server address being displayed incorrectly in the CLI when viewing init logs.
I also explicitly added the `:latest` tag for the Docker image in the Running example to make it more clear which tag is being run (no tag defaults to `latest`.)
* xftp: expire files on the server
* track/limit used storage
* support storage quota and disabling queue creation in CLI parameters
* fix ini file
* correction
---------
Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
* server_key_hash fields
* test
* refactor
* fix
* order
* use sync command in test
* refactor
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* xftp: cli client draft
* more stubs
* compiles
* hash, app
* options parsers, random
* tmp
* xftp CLI client agent, simplify CLI command syntax
* only allow argument as a second parameter
* pivot signature draft
* receive file
* pivot sent chunks to recipients
* encryptFile - temp, chunks, specs
* send (upload) file and save file descriptions
* refactor, remove encrypted file
* save file size in description as string
* include filename inside padded encrypted file
* call chunk uploads concurrently, using queueing in HTTP2 as library client does not support concurrent streaming uploads
* download file (does not work yet)
* add digests to sent chunks
* fix recv - save file using AppendMode
* encrypt/decrypt sent file with secretbox
* remove print
* fix file description parsing in tests
* fix test
---------
Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
* end SMP client connection after configured number of PING errors, reset PING error count on any success
* only terminate client on PING timeout
* refactor
* comment
* execute asynchronous commands for correct users
* pass transport session to message processing to avoid race condition
* account for server changes when determining unused servers
* enable one test
* enable all tests, remove log
* subscribe users in different sessions
* remove comments
* include userId to rcv queue map key
* use hash of userId[:entityId] as SOCKS proxy username
Add `XFTPClientAgent` — a per-server connection pool matching the Haskell pattern. The agent caches `XFTPClient` instances by server URL. All orchestration functions (`uploadFile`, `downloadFile`, `deleteFile`) take `agent` as first parameter and use `getXFTPServerClient(agent, server)` instead of calling `connectXFTP` directly. Connections stay open on success; the caller creates and closes the agent.
`connectXFTP` and `closeXFTP` stay exported (used by `XFTPWebTests.hs` Haskell tests). The `browserClients` hack, per-function `connections: Map`, and `getOrConnect` are deleted.
## Changes: client.ts
**Add** after types section: `XFTPClientAgent` interface, `newXFTPAgent`, `getXFTPServerClient`, `closeXFTPServerClient`, `closeXFTPAgent`.
**Delete**: `browserClients` Map and all `isNode` browser-cache checks in `connectXFTP` and `closeXFTP`.
**Revert `closeXFTP`** to unconditional `c.transport.close()` (browser transport.close() is already a no-op).
`connectXFTP` stays exported (backward compat) but becomes a raw low-level function — no caching.
## Changes: agent.ts
**Imports**: replace `connectXFTP`/`closeXFTP` with `getXFTPServerClient`/`closeXFTPAgent` etc.
**Re-export** from agent.ts: `newXFTPAgent`, `closeXFTPAgent`, `XFTPClientAgent`.
**`uploadFile`**: add `agent: XFTPClientAgent` as first param. Replace `connectXFTP` → `getXFTPServerClient`. Remove `finally { closeXFTP }`. Pass `agent` to `uploadRedirectDescription`.
**`uploadRedirectDescription`**: change from `(client, server, innerFd)` to `(agent, server, innerFd)`. Get client via `getXFTPServerClient`.
**`downloadFile`**: add `agent` param. Delete local `connections: Map`. Replace `getOrConnect` → `getXFTPServerClient`. Remove finally cleanup. Pass `agent` to `downloadWithRedirect`.
- allow receiving multiple messages from single iOS notification (#1355, #1362).
- prepare connection to accept to avoid race condition with events (#1365).
- transport isolation mode "Session" (default) to use new SOCKS credentials when client restarts or SOCKS proxy configuration changes (#1321).
Ntf server:
- control port (#1354).
- enable pings on ntf subscriptions, to resubscribe on reconnection (#1353).
SMP server:
- support multiple server ports (#1319).
- support serving HTTPS and SMP transport on the same port (#1326, #1327).
- persist iOS notifications to avoid losing them when Ntf server is offline (#1336, #1339, #1350).
- fix lost notification subscriptions (#1347).
- reject SKEY with different key earlier, at verification step (#1366).
- pass server information via CLI during server initialization (#1356).
- show version on server page (#1341).
- explicit graceful shutdown on SIGINT (#1360).
XRCP (remote access protocol):
- use SHA3-256 in hybrid key agreement (#1302).
- session encryption with forward secrecy (#1328).
# 6.0.5
SMP agent:
- support generic SOCKS proxy (without isolate-by-auth).
- reduce max message sizes
# 6.0.4
SMP server:
- better performance/memory: fewer map updates on re-subscriptions (#1297), split and reduce STM transactions (#1294)
- send DELD when subscribed queue is deleted (#1312)
- add created/updated/used date to queues to manage expiration (#1306)
XFTP server: truncate file creation time to 1 hour (#1310)
Servers:
- bind control port only to 127.0.0.1 for better security in case of firewall misconfiguration (#1280)
- reduce memory used for period stats (#1298)
Agent: process last notification from list (#1307)
- report receive file error with redirected file ID, when redirect is present (#1304)
- special error when deleted user record is not in database (#1303)
- fix race when sending a message to the deleted connection (#1296)
- support for multiple messages in a single notification
Ntf server:
- only use SOCKS proxy for servers without public address (#1314)
# 6.0.3
Agent:
- fix possible stuck queue rotation (#1290).
SMP server:
- batch END responses when subscribed client switches to reduce server and client traffic.
- reduce STM transactions for better performance.
- add stats for END events and for SUB/DEL event batches.
- remove "expensive" stats to save memory.
# 6.0.2
SMP agent:
- fix stuck connection commands when a server is not responding.
- store query errors, reduce slow query threshold to 1ms.
Notification server:
- reduce PING interval to 1 minute.
- fix subscriptions disabled on race condition (only mark subscriptions with END status when received via the active connection).
# 6.0.1
SMP agent:
- support changing user of the new connection.
- do not start delivery workers when there are no messages to deliver.
- enable notifications for all connections.
- combine database transactions when subscribing.
SMP server:
- safe compacting of store log.
- fix possible race when creating client that might lead to memory leak.
Dependencies: upgrade tls to 1.9
# 6.0.0
Version 6.0.0.8
Agent:
- enabled fast handshake support.
- batch-send multiple messages in each connection.
- resume subscriptions as soon as agent moves to foreground or as network connection resumes.
- "known" servers to determine whether to use SMP proxy.
- retry on SMP proxy NO_SESSION error.
- fixes to notification subscriptions.
- persistent server statistics.
- better concurrency.
SMP server:
- reduce threads usage.
- additional statistics.
- improve disabling inactive clients.
- additional control port commands for monitoring.
Notification server:
- support onion-only SMP servers.
# 5.8.2
Agent:
- fast handshake support (disabled).
- new statistics api.
SMP server:
- fast handshake support (SKEY command).
- minor changes to reduce memory usage.
# 5.8.1
Agent:
- API to reconnect one server.
- Better error handling of file errors and remote control connection errors.
- Only start uploading file once all chunks were registered on the servers.
SMP server:
- additional stats for sent message notifications.
- fix server page layout.
# 5.8.0
Version 5.8.0.10
SMP server and client:
- protocol extension to forward messages to the destination servers, to protect sending client IP address and transport session.
Agent:
- process timed out subscription responses to reduce the number of resubscriptions.
- avoid sending messages and commands when waiting for response timed out (except batched SUB and DEL commands).
- fix issue with stuck message reception on slow connection (when response to ACK timed out, and the new message was not processed until resubscribed).
- fix issue when temporary file sending or receiving error was treated as permanent.
SMP server:
- include OK responses to all batched SUB requests to reduce subscription timeouts.
XFTP server:
- report file upload timeout as TIMEOUT, to avoid delivery failure.
# 5.7.6
XFTP agent:
- treat XFTP handshake timeouts and network errors as temporary, to retry file operations.
# 5.7.5
SMP agent:
- fail if non-unique connection IDs are passed to sendMessages (to prevent client errors and deadlocks).
# 5.7.4
SMP agent:
- remove re-subscription timeouts (as they are tracked per operation, and could cause failed subscriptions).
- reconnect XFTP clients when network settings changes.
- fix lock contention resulting in stuck subscriptions on network change.
# 5.7.3
SMP/NTF protocol:
- add ALPN for handshake version negotiation, similar to XFTP (to preserve backwards compatibility with the old clients).
- upgrade clients to versions v7/v2 of the protocols.
SMP server:
- faster responses to subscription requests.
XFTP client:
- fix network exception during file download treated as permanent file error.
SMP agent:
- do not report subscription timeouts while client is offline.
# 5.7.2
SMP agent:
- fix connections failing when connecting via link due to race condition on slow network.
- remove concurrency limit when waiting for connection subscription.
- remove TLS timeout.
# 5.7.1
SMP agent:
- increase timeout for TLS connection via SOCKS
# 5.7.0
Version 5.7.0.4
_Please note_: the earliest SimpleX Chat clients supported by this version of the servers is 5.5.3 (released on February 11, 2024).
SMP server:
- increase max SMP protocol version to 7 (support for deniable authenticators).
NTF server:
- increase max NTF protocol version to 2 (support for deniable authenticators).
XFTP server:
- version handshake using ALPN.
SMP agent:
- increase timeouts for XFTP files.
- don't send commands after timeout.
- PQ encryption support.
# 5.6.2
Version 5.6.2.2.
SMP agent:
- Lower memory consumption (~20-25%).
- More stable XFTP file uploads and downloads.
- API to receive network connectivity changes from the apps.
- to reduce battery consumption: connection attempts interval growing to every 2 hours when app reports as offline.
- to reduce retries and traffic: 50% increased timeouts when on mobile network.
XFTP server:
- expire files on start.
- version negotiation based on TLS ALPN and handshake.
NTF server:
- reduced downtime by ~100x faster start time.
- exclude test tokens from statistics.
# 5.6.1
Version 5.6.1.0.
- Much faster iOS notification server start time (fewer skipped notifications).
- Fix SMP server stored message stats.
- Prevent overwriting uploaded XFTP files with subsequent upload attempts.
- Faster base64 encoding/parsing.
- Control port audit log and authentication.
# 5.6.0
Version 5.6.0.4.
SMP protocol/client/server:
- support deniable sender command authorization (to be enabled in the next version).
- remove support for SMP protocol versions (prior to v4, 07/2022).
Agent:
- optional post-quantum key agreement using sntrup761 in double ratchet protocol.
- improve performance of deleting multiple connections and files by batching database operations.
- delay connection deletion to deliver pending messages.
- API to test for notifications server.
- remove support for client protocols versions (prior to 10/2022).
XFTP server:
- restore storage quota in case of failed uploads.
Performance and stability improvements.
# 5.5.3
Agent:
- notification token API also returns active notifications server.
- support file descriptions with redirection and file URIs.
Servers:
- CLI commands for online key and certificate rotation.
- Configure config and log paths via environment variables.
# 5.5.2
Extensible handshake for clients and SMP/NTF servers (ignore extra data).
# 5.5.1
SMP servers:
- do not keep stats file open
- additional stats about currently stored messages
Agent:
- support multiple notification servers (only one can be used at a time).
- expire messages after "quota exceeded" error after 7 days (instead of 21 days previously).
- stabilize message delivery, remove unnecessary subscription retries and traffic.
- improve database performance for message delivery.
- fix sockets/memory leak - a very old bug "activated" by improvements in v5.5.0.
# 5.5.0
Code:
- compatible with GHC 8.10.7 to support compilation for armv7a.
- migrate to `crypton` from deprecated `cryptonite` (the seed for DRG is now sha512-hashed).
- use ChaChaDRG for all random IDs, keys and nonces, only using hashed entropy as seed.
- more efficient transaction batching in SMP protocol client and server.
Agent:
- stabilize message reception and delivery, migrate message delivery to database queue.
- additional event MSGNTF confirming that message received via notification is processed.
- efficient processing of messages sent to multiple recipients with batched database transactions.
- new worker abstraction for all queued tasks resilient to race conditions and some database errors.
- many fixed race conditions.
- background mode for iOS NSE.
- additional error reporting to client on critical errors (to be show as alert in the clients).
- functional api to get worker statistics.
SMP/XFTP servers:
- fix socket and memory leak on servers with high load (inactive clients without subscriptions are disconnected after set time of inactivity).
- control port improvements.
- fix statistics for stored queues, messages and files.
- make writing to store log atomic (fixes a rare bug in XFTP server).
- more robust connection switching logic, API to abort switching the address
Notification server:
- batch subscriptions to SMP servers
# 5.1.1 (NTF server 1.4.0)
Agent:
- store and check hashes of previous encrypted messages to differentiate between duplicates and decryption errors
Server:
- larger processing queues
- expire messages when restoring them
# 5.1.0
XFTP client:
- check encrypted file exists when uploading
- remove user ID from deletion API
Agent:
- vacuum database on migrations
SMP server:
- configure message expiration time in INI file
# 5.0.0
SimpleX File Transfer Protocol (XFTP):
- XFTP server
- XFTP CLI
- support XFTP in the agent
Other changes:
- preserve retry interval for sending messages on agent restarts
- support IPv6 in the servers and in the agent
- support for 32-bit platforms
Read more about XFTP in this post: https://simplex.chat/blog/20230301-simplex-file-transfer-protocol.html
# 4.4.1
SMP agent:
- fix handling of server addresses.
- prevent message delivery getting stuck on IO error in SMP client.
# 4.4.0
SMP agent:
- support multiple user profiles with transport session isolation.
- support optional transport isolation per connection.
- batch connection deletion
- improve asynchronous connection deletion – it may now be completed after the client is restarted as well.
- improve subscription logic to retry if initial attempt fails.
- end SMP client connection after a number of failed PINGs (default is 3).
# 4.3.0
SMP server:
- additional server usage statistics.
SMP agent:
- increase retry interval when sending messages after ERR QUOTA.
# 4.2.0
SMP agent and server:
@@ -19,7 +650,7 @@ SMP agent:
Notification server (v1.3.0):
- check token status when sending verification notificaiton.
- check token status when sending verification notification.
# 4.1.0
@@ -38,15 +669,15 @@ SMP agent:
SMP server:
- Basic authentication. The server address can now include an optional password that is required to create messaging queues, so the contacts who message you will not be able to receive messages via your server, unless you share with them the address with the password. It is recommended to enable basic authentication on all private servers by adding `create_password` parameter into AUTH section of server INI file (the previously deployed servers do not have this section, you need to add it).
- Disable creating new queues completely with `new_queues: off` parameter in AUTH section of INI file - it can be used to simplify migrating the exising connections to another server.
- Disable creating new queues completely with `new_queues: off` parameter in AUTH section of INI file - it can be used to simplify migrating the existing connections to another server.
- Updated server CLI with changed defaults:
- interactive server initialization, use -y flag to initalize the server non-interactively.
- interactive server initialization, use -y flag to initialize the server non-interactively.
- store log is now enabled by default, so messaging queues and messages are restored when the server is restarted (to restore undelivered messages the server needs to be stopped with SIGINT signal).
- a random password is now generated by default during the server initialization.
SMP agent:
- API to test SMP servers. It connects to the server, creates and deletes a messaging queue. The new SimpleX Chat client uses this API to allow you to test that you have the correct server address, with the valid certificate fingerprint and pasword, before enabling the new server.
- API to test SMP servers. It connects to the server, creates and deletes a messaging queue. The new SimpleX Chat client uses this API to allow you to test that you have the correct server address, with the valid certificate fingerprint and password, before enabling the new server.
# 3.4.0
@@ -92,9 +723,9 @@ SMP server and agent:
SMP server:
- restore undeliverd messages when the server is restarted.
- restore undelivered messages when the server is restarted.
- SMP protocol v3 to support push notification:
- updated SEND and MSG to add message flags (for notification flag that contros whether the notification is sent and for any future extensions) and to move message meta-data sent to the recipient into the encrypted envelope.
- updated SEND and MSG to add message flags (for notification flag that confirm whether the notification is sent and for any future extensions) and to move message meta-data sent to the recipient into the encrypted envelope.
- update NKEY and NID to add e2e encryption keys (for the notification meta-data encryption between SMP server and the client), and update NMSG to include this meta-data.
- update ACK command to include message ID (to avoid acknowledging unprocessed message).
- add NDEL commands to remove notification subscription credentials from SMP queue.
@@ -105,7 +736,7 @@ SMP agent:
- new protocol for duplex connection handshake reducing traffic and connection time.
- support for SMP notifications server and managing device token.
- remove redundant FQDN validation from TLS handshake to prepare for access via Tor.
- support for fully stopping agent and for termporary suspending agent operations.
- support for fully stopping agent and for temporary suspending agent operations.
- improve management of duplicate message delivery.
📢 SimpleXMQ v1 is released - with many security, privacy and efficiency improvements, new functionality - see [release notes](https://github.com/simplex-chat/simplexmq/releases/tag/v1.0.0).
@@ -33,7 +33,7 @@ To initialize the server use `smp-server init -n <fqdn>` (or `smp-server init --
SMP server uses in-memory persistence with an optional append-only log of created queues that allows to re-start the server without losing the connections. This log is compacted on every server restart, permanently removing suspended and removed queues.
To enable store log, initialize server using `smp-server -l` command, or modify `smp-server.ini` created during initialization (uncomment `enable: on` option in the store log section). Use `smp-server --help` for other usage tips.
To enable store log, initialize server using `smp-server -l` command, or modify `smp-server.ini` created during initialization (uncomment `enable = on` option in the store log section). Use `smp-server --help` for other usage tips.
Starting from version 2.3.0, when store log is enabled, the server would also enable saving undelivered messages on exit and restoring them on start. This can be disabled via a separate setting `restore_messages` in `smp-server.ini` file. Saving messages would only work if the server is stopped with SIGINT signal (keyboard interrupt), if it is stopped with SIGTERM signal the messages would not be saved.
@@ -90,11 +90,11 @@ You can either run your own SMP server locally or deploy using [Linode StackScri
It's the easiest to try SMP agent via a prototype [simplex-chat](https://github.com/simplex-chat/simplex-chat) terminal UI.
## Deploy SMP server on Linux
## Deploy SMP/XFTP servers on Linux
You can run your SMP server as a Linux process, optionally using a service manager for booting and restarts.
You can run your SMP/XFTP server as a Linux process, optionally using a service manager for booting and restarts.
Notice that `smp-server` requires `openssl` as run-time dependency (it is used to generate server certificates during initialization). Install it with your packet manager:
Notice that `smp-server` and `xftp-server` requires `openssl` as run-time dependency (it is used to generate server certificates during initialization). Install it with your packet manager:
- Initialize SMP server with `smp-server init [-l] -n <fqdn>` or `smp-server init [-l] --ip <ip>` - depending on how you initialize it, either FQDN or IP will be used for server's address.
d="m 282.715,299.835 a 3.29,3.29 0 0 0 -2.662,5.336 l 9.474,12.261 A 7.894,7.894 0 0 0 289,320.257 v 18.21 a 7.877,7.877 0 0 0 7.895,7.895 h 84.21 A 7.877,7.877 0 0 0 389,338.468 v -18.211 c 0,-0.999 -0.19,-1.949 -0.525,-2.826 l 9.472,-12.26 a 3.29,3.29 0 0 0 -2.433,-5.334 3.29,3.29 0 0 0 -2.772,1.31 l -9.013,11.666 a 7.91,7.91 0 0 0 -2.624,-0.45 h -84.21 c -0.922,0 -1.8,0.163 -2.622,0.45 l -9.015,-11.666 a 3.29,3.29 0 0 0 -2.543,-1.312 z m 14.18,49.527 A 7.877,7.877 0 0 0 289,357.257 v 52.21 a 7.877,7.877 0 0 0 7.895,7.895 h 84.21 A 7.877,7.877 0 0 0 389,409.468 v -52.211 a 7.877,7.877 0 0 0 -7.895,-7.895 z"
<textx="-6"y="240"text-anchor="middle"font-family="system-ui, sans-serif"font-size="10"fill="#70F0F9"transform="rotate(-90 -6 240)">key in URL fragment - never sent to page server or data router</text>
<!-- Closed padlock: encryption (between sender and chunks) -->
<textx="-6"y="240"text-anchor="middle"font-family="system-ui, sans-serif"font-size="10"fill="#0053D0"transform="rotate(-90 -6 240)">key in URL fragment - never sent to page server or data router</text>
<!-- Closed padlock: encryption (between sender and chunks) -->
The implementation of sntrup761 is the _exact_ copy from this [Internet draft](https://www.ietf.org/archive/id/draft-josefsson-ntruprime-streamlined-00.html).
This file provides guidance on coding style and approaches and on building the code.
## Code Security
When designing code and planning implementations:
- Apply adversarial thinking, and consider what may happen if one of the communicating parties is malicious.
- Formulate an explicit threat model for each change - who can do which undesirable things and under which circumstances.
## Code Quality Standards
Haskell client and server code serves as system specification, not just implementation — we use type-driven design to reflect the business domain in types. Quality, conciseness, and clarity of Haskell code are critical.
## Code Style, Formatting and Approaches
The project uses **fourmolu** for Haskell code formatting. Configuration is in `fourmolu.yaml`.
**Key formatting rules:**
- 2-space indentation
- Trailing function arrows, commas, and import/export style
- Record brace without space: `{field = value}`
- Single newline between declarations
- Never use unicode symbols
- Inline `let` style with right-aligned `in`
**Format code before committing:**
```bash
# Format a single file
fourmolu -i src/Simplex/Messaging/Protocol.hs
```
Some files that use CPP language extension cannot be formatted as a whole, so individual code fragments need to be formatted.
**Follow existing code patterns:**
- Match the style of surrounding code
- Use qualified imports with short aliases (e.g., `import qualified Data.ByteString.Char8 as B`)
- Use record syntax for types with multiple fields
- Prefer explicit pattern matching over partial functions
**Comments policy:**
- Avoid redundant comments that restate what the code already says
- Only comment on non-obvious design decisions or tricky implementation details
- Function names and type signatures should be self-documenting
- Do not add comments like "wire format encoding" (Encoding class is always wire format) or "check if X" when the function name already says that
- Assume a competent Haskell reader
**Diff and refactoring:**
- Avoid unnecessary changes and code movements
- Never do refactoring unless it substantially reduces cost of solving the current problem, including the cost of refactoring
- Aim to minimize the code changes - do what is minimally required to solve users' problems
**Document and code structure:**
- **Never move existing code or sections around** - add new content at appropriate locations without reorganizing existing structure.
- When adding new sections to documents, continue the existing numbering scheme.
- Trace data flows end-to-end: from origin, through storage/parameters, to consumption. Flag values that are discarded and reconstructed from partial data (e.g. extracted from a URI missing original fields) — this is usually a bug.
- Read implementations of called functions, not just signatures — if duplication involves a called function, check whether decomposing it resolves the duplication.
- Do not save time on analysis. Read every function in the data flow even when the interface seems clear — wrong assumptions about internals are the main source of missed bugs.
### Haskell Extensions
- `StrictData` enabled by default
- Use STM for safe concurrency
- Assume concurrency in PostgreSQL queries
- Comprehensive warning flags with strict pattern matching
- **SMP Server**: Message broker with TLS, in-memory queues, optional persistence ([main code](../src/Simplex/Messaging/Server.hs), [all code files](../src/Simplex/Messaging/Server/), [executable](../apps/smp-server/)). For proxying SMP commands the server uses [lightweight SMP client](../src/Simplex/Messaging/Client/Agent.hs).
- **SMP Client**: Functional API with STM-based message delivery ([code](../src/Simplex/Messaging/Client.hs)).
- **SMP Agent**: High-level duplex connections via multiple simplex queues with E2E encryption ([code](../src/Simplex/Messaging/Agent.hs)). Implements Agent-to-agent protocol ([code](../src/Simplex/Messaging/Agent/Protocol.hs), [spec](../protocol/agent-protocol.md)) via intermediary agent client ([code](../src/Simplex/Messaging/Agent/Client.hs)).
- **XFTP**: SimpleX File Transfer Protocol, server and CLI client ([code](../src/Simplex/FileTransfer/), [spec](../protocol/xftp.md)).
- **XRCP**: SimpleX Remote Control Protocol ([code](`../src/Simplex/RemoteControl/`), [spec](../protocol/xrcp.md)).
- **Notifications**: Push notifications server requires PostgreSQL ([code](../src/Simplex/Messaging/Notifications), [executable](../apps/ntf-server/)). Client protocol is used for clients to communicate with the server ([code](../src/Simplex/Messaging/Notifications/Protocol.hs), [spec](../protocol/push-notifications.md)). For subscribing to SMP notifications the server uses [lightweight SMP client](../src/Simplex/Messaging/Client/Agent.hs).
## Architecture
For general overview see `../protocol/overview-tjr.md`.
We do not make code changes to improve code - any change must address a specific user problem or request.
## Discuss the plans as early as possible
Please discuss the problem you want to solve and your detailed implementation plan with the project team prior to contributing, to avoid wasted time and additional changes. Acceptance of your contribution depends on your willingness and ability to iterate the proposed contribution to achieve the required quality level, coding style, test coverage, and alignment with user requirements as they are understood by the project team.
## Follow project structure, coding style and approaches
./PROJECT.md has information about the structure of this `simplexmq` repository.
./CODE.md has details about general requirements common for `simplexmq` and `simplex-chat` repositories.
This files can be used with LLM prompts, e.g. if you use Claude Code you can create CLAUDE.md file in project root importing content from these files:
*)printf"${RED}Unsupported Ubuntu version!${NC}\nPlease file Github issue with request to support Ubuntu %s: https://github.com/simplex-chat/simplexmq/issues/new\n""$VERSION_ID"&&exit1;;
esac
version="$(printf'%s'"$VERSION_ID"| tr '.''_')"
arch="$(uname -p)"
case"$arch" in
x86_64)arch="$(printf'%s'"$arch"| tr '_''-')";;
*)printf"${RED}Unsupported architecture!${NC}\nPlease file Github issue with request to support %s architecture: https://github.com/simplex-chat/simplexmq/issues/new""$arch"&&exit1;;
Add PostgreSQL backend support to xftp-server, following the SMP server pattern. Supports bidirectional migration between STM (in-memory with StoreLog) and PostgreSQL backends.
## Goals
- PostgreSQL-backed file metadata storage as an alternative to STM + StoreLog
- Polymorphic server code via `FileStoreClass` typeclass with IO-based methods (following `QueueStoreClass` pattern)
- Bidirectional migration: StoreLog <-> PostgreSQL via CLI commands
- Shared `server_postgres` cabal flag (same flag enables both SMP and XFTP Postgres support)
- INI-based backend selection at runtime
## Architecture
### FileStoreClass Typeclass
IO-based typeclass following the `QueueStoreClass` pattern — each method is a self-contained IO action, with the implementation responsible for its own atomicity (STM backend wraps in `atomically`, Postgres backend uses database transactions):
deleteRecipient :: s -> RecipientId -> FileRec -> IO ()
ackFile :: s -> RecipientId -> IO (Either XFTPErrorType ())
-- Expiration (with LIMIT for Postgres; called in a loop until empty)
expiredFiles :: s -> Int64 -> Int -> IO [(SenderId, Maybe FilePath, Word32)]
-- Storage and stats (for init-time computation)
getUsedStorage :: s -> IO Int64
getFileCount :: s -> IO Int
```
- STM backend: each method wraps its STM transaction in `atomically` internally.
- Postgres backend: each method runs its query via `withDB` / database connection internally.
No polymorphic monad or `runStore` dispatcher needed — unlike `MsgStoreClass`, XFTP file operations are individually atomic and don't require grouping multiple operations into backend-dependent transactions.
### PostgresFileStore Data Type
```haskell
data PostgresFileStore = PostgresFileStore
{ dbStore :: DBStore,
dbStoreLog :: Maybe (StoreLog 'WriteMode)
}
```
- `dbStore` — connection pool created via `createDBStore`, runs schema migrations on init.
- `dbStoreLog` — optional parallel log file (enabled by `db_store_log` INI setting). When present, every mutation (`addFile`, `setFilePath`, `deleteFile`, `blockFile`, `addRecipient`, `ackFile`) also writes to this log via a `withLog` wrapper. `withLog` is called AFTER the DB operation succeeds (so the log reflects committed state only). Log write failures are non-fatal (logged as warnings, do not fail the DB operation). This provides an audit trail and enables recovery via export.
`closeFileStore` for Postgres calls `closeDBStore` (closes connection pool) then `mapM_ closeStoreLog dbStoreLog` (flushes and closes the parallel log). For STM, it closes the storeLog. Called from a `finally` block during server shutdown, matching SMP's `stopServer` → `closeMsgStore` → `closeQueueStore` pattern.
### STMFileStore Type
After extracting from current `Store.hs`, `STMFileStore` retains the file and recipient maps but no longer owns `usedStorage` (moved to `XFTPEnv`):
withDB :: Text -> PostgresFileStore -> (DB.Connection -> IO (Either XFTPErrorType a)) -> ExceptT XFTPErrorType IO a
withDB op st action =
ExceptT $ E.try (withTransaction (dbStore st) action) >>= either logErr pure
where
logErr :: E.SomeException -> IO (Either XFTPErrorType a)
logErr e = logError ("STORE: " <> err) $> Left INTERNAL
where
err = op <> ", withDB, " <> tshow e
handleDuplicate :: SqlError -> IO (Either XFTPErrorType a)
handleDuplicate e = case constraintViolation e of
Just (UniqueViolation _) -> pure $ Left DUPLICATE_
_ -> E.throwIO e
```
- All DB operations wrapped in `withDB` — catches exceptions, logs, returns `INTERNAL`.
- Unique constraint violations caught by `handleDuplicate` and mapped to `DUPLICATE_`.
- UPDATE operations verified with `assertUpdated` — returns `AUTH` if 0 rows affected (matching SMP pattern, prevents silent failures when WHERE clause doesn't match).
- Critical sections (DB write + TVar update) wrapped in `uninterruptibleMask_` to prevent async exceptions from leaving inconsistent state between DB and TVars.
### FileRec and TVar Fields
`FileRec` retains its `TVar` fields (matching SMP's `PostgresQueue` pattern):
```haskell
data FileRec = FileRec
{ senderId :: SenderId,
fileInfo :: FileInfo,
filePath :: TVar (Maybe FilePath),
recipientIds :: TVar (Set RecipientId),
createdAt :: RoundedFileTime,
fileStatus :: TVar ServerEntityStatus
}
```
- **STM backend**: TVars are the source of truth, as currently.
- **Postgres backend**: `getFile` reads from DB and creates a `FileRec` with fresh TVars populated from the DB row (matching SMP's `mkQ` pattern — `newTVarIO` per load). Mutation methods (`setFilePath`, `blockFile`, etc.) update both the DB (persistence) and the TVars (in-session consistency). The `recipientIds` TVar is initialized to `S.empty` — no subquery needed because no server code reads `recipientIds` directly; all recipient operations go through the typeclass methods (`addRecipient`, `deleteRecipient`, `ackFile`), which query the `recipients` table for Postgres.
### usedStorage Ownership
`usedStorage :: TVar Int64` moves from the store to `XFTPEnv`. The store typeclass does **not** manage `usedStorage` — it only provides `getUsedStorage` for init-time computation.
- **STM init**: StoreLog replay calls `setFilePath` (which only sets the filePath TVar — the STM `setFilePath` implementation is changed to **not** update `usedStorage`). Similarly, STM `deleteFile` (Store.hs line 117) and `blockFile` (line 125) are changed to **not** update `usedStorage` — the server handles all `usedStorage` adjustments externally. After replay, `getUsedStorage` computes the sum over all file sizes (matching current `countUsedStorage` behavior).
- **Postgres init**: `getUsedStorage` executes `SELECT COALESCE(SUM(file_size), 0) FROM files`.
- **Runtime**: Server manages `usedStorage` TVar directly for reserve/commit/rollback during uploads, and adjusts after `deleteFile`/`blockFile` calls.
**Note on `getUsedStorage` semantics**: The current STM `countUsedStorage` sums all file sizes unconditionally (including files without `filePath` set, i.e., created but not yet uploaded). The Postgres `getUsedStorage` matches this: `SELECT SUM(file_size) FROM files` (no `WHERE file_path IS NOT NULL`). In practice, orphaned files (created but never uploaded) are rare and short-lived (expired within 48h), so the difference is negligible. A future improvement could filter by `file_path IS NOT NULL` in both backends to reflect actual disk usage more accurately.
### Server.hs Refactoring
`Server.hs` becomes polymorphic over `FileStoreClass s`. Since all typeclass methods are IO, call sites replace `atomically` with direct IO calls to the store.
1. **`receiveServerFile`** (line 563): `atomically $ writeTVar filePath (Just fPath)` → `setFilePath store senderId fPath`. The `reserve` logic (line 551-555) stays as direct TVar manipulation on `usedStorage` from `XFTPEnv`.
2. **`verifyXFTPTransmission`** (line 453): `atomically $ verify =<< getFile st party fId` — the `getFile` call and subsequent `readTVar fileStatus` are in a single `atomically` block. Refactored to: `getFile st party fId` (IO), then `readTVarIO (fileStatus fr)` from the returned `FileRec` (safe for both backends — STM TVar is the source of truth, Postgres TVar is a fresh snapshot from DB).
4. **`deleteOrBlockServerFile_`** (line 620): Parameter `FileStore -> STM (Either XFTPErrorType ())` → `FileStoreClass s => s -> IO (Either XFTPErrorType ())`. The `atomically` call (line 626) removed — the store method is already IO. After the store action, server adjusts `usedStorage` TVar in `XFTPEnv` based on `fileInfo.size`.
5. **`ackFileReception`** (line 605): `atomically $ deleteRecipient st rId fr` → `deleteRecipient st rId fr`.
7. **`expireServerFiles`** (line 636): Replace per-file `expiredFilePath` iteration with batched `expiredFiles st old batchSize`, which returns `[(SenderId, Maybe FilePath, Word32)]` — the `Word32` file size is needed so the server can adjust the `usedStorage` TVar after each deletion. Called in a loop until the returned list is empty. The `itemDelay` between files applies to the deletion loop over each batch, not the query itself. STM backend ignores the batch size limit (returns all expired files from TMap scan); Postgres uses `LIMIT`.
8. **`restoreServerStats`** (line 694): `FileStore {files, usedStorage} <- asks store` accesses store fields directly. Refactored to: `usedStorage` from `XFTPEnv` via `asks usedStorage`, file count via `getFileCount store`. STM: `M.size <$> readTVarIO files`. Postgres: `SELECT COUNT(*) FROM files`.
The `M` monad (`ReaderT (XFTPEnv s) IO`) and all functions in `Server.hs` gain `FileStoreClass s =>` constraints.
**StoreLog lifecycle per backend:**
- **STM mode**: `storeLog = Just sl` (current behavior — append-only log for persistence and recovery).
- **Postgres mode**: `storeLog = Nothing` (main storeLog disabled — Postgres is the source of truth). The optional parallel `dbStoreLog` inside `PostgresFileStore` provides audit/recovery if enabled via `db_store_log` INI setting.
The existing `withFileLog` pattern in Server.hs continues to work unchanged — it maps over `Maybe (StoreLog 'WriteMode)`, which is `Nothing` in Postgres mode so the calls become no-ops.
### Main.hs Store Type Dispatch
The `Start` CLI command gains a `--confirm-migrations` flag (default `MCConsole` — manual prompt, matching SMP's `StartOptions`). For automated deployments, `--confirm-migrations up` auto-applies forward migrations. The import command uses `MCYesUp` (always auto-apply).
Following SMP's existential dispatch pattern (`AStoreType` + `run`), `Main.hs` selects the store type from INI config and dispatches to the polymorphic server:
```haskell
runServer ini = do
let storeType = fromRight "memory" $ lookupValue "STORE_LOG" "store_files" ini
case storeType of
"memory" -> run $ XSCMemory (enableStoreLog $> storeLogFilePath)
"database" ->
#if defined(dbServerPostgres)
run $ XSCDatabase PostgresFileStoreCfg {..}
#else
exitError "server not compiled with Postgres support"
run :: FileStoreClass s => XFTPStoreConfig s -> IO ()
run storeCfg = do
env <- newXFTPServerEnv storeCfg config
runReaderT (xftpServer config) env
```
**`newXFTPServerEnv` refactored signature:**
```haskell
newXFTPServerEnv :: FileStoreClass s => XFTPStoreConfig s -> XFTPServerConfig -> IO (XFTPEnv s)
newXFTPServerEnv storeCfg config = do
(store, storeLog) <- case storeCfg of
XSCMemory storeLogPath -> do
st <- newFileStore ()
sl <- mapM (`readWriteFileStore` st) storeLogPath
pure (st, sl)
XSCDatabase dbCfg -> do
st <- newFileStore dbCfg
pure (st, Nothing) -- main storeLog disabled for Postgres
usedStorage <- newTVarIO =<< getUsedStorage store
...
pure XFTPEnv {config, store, usedStorage, storeLog, ...}
```
### Startup Config Validation
Following SMP's `checkMsgStoreMode` pattern, `Main.hs` validates config before starting:
- **`store_files=database` + StoreLog file exists** (without `db_store_log=on`): Error — "StoreLog file present but store_files is `database`. Use `xftp-server database import` to migrate, or set `db_store_log: on`."
- **`store_files=database` + schema doesn't exist**: Error — "Create schema in PostgreSQL or use `xftp-server database import`."
- **`store_files=memory` + Postgres schema exists**: Warning — "Postgres schema exists but store_files is `memory`. Data in Postgres will not be used."
- **Binary compiled without `server_postgres` + `store_files=database`**: Error — "Server not compiled with Postgres support."
Main.hs -- Store selection, migration CLI commands
Server.hs -- Polymorphic over FileStoreClass
```
## PostgreSQL Schema
Initial migration (`20260325_initial`):
```sql
CREATE TABLE files (
sender_id BYTEA NOT NULL PRIMARY KEY,
file_size INT4 NOT NULL,
file_digest BYTEA NOT NULL,
sender_key BYTEA NOT NULL,
file_path TEXT,
created_at INT8 NOT NULL,
status TEXT NOT NULL DEFAULT 'active'
);
CREATE TABLE recipients (
recipient_id BYTEA NOT NULL PRIMARY KEY,
sender_id BYTEA NOT NULL REFERENCES files ON DELETE CASCADE,
recipient_key BYTEA NOT NULL
);
CREATE INDEX idx_recipients_sender_id ON recipients (sender_id);
CREATE INDEX idx_files_created_at ON files (created_at);
```
- `file_size` is `INT4` matching `Word32` in `FileInfo.size`
- `sender_key` and `recipient_key` stored as `BYTEA` using binary encoding via `C.encodePubKey` / `C.decodePubKey` (matching SMP's `ToField`/`FromField` instances for `APublicAuthKey` — includes algorithm type tag in the binary format)
- `file_path` nullable (set after upload completes via `setFilePath`)
- `ON DELETE CASCADE` for recipients when file is hard-deleted
- `status` as TEXT via `StrEncoding` (`ServerEntityStatus`: `EntityActive`, `EntityBlocked info`, `EntityOff`)
- Hard deletes (no `deleted_at` column)
- No PL/pgSQL functions needed; `setFilePath` uses `WHERE file_path IS NULL` to prevent duplicate uploads (the `UPDATE` itself acquires a row-level lock)
- `used_storage` computed on startup: `SELECT COALESCE(SUM(file_size), 0) FROM files` (matches STM `countUsedStorage` — all files, see usedStorage Ownership section)
### Migrations Module
Following SMP's `QueueStore/Postgres/Migrations.hs` pattern:
The `Migration` type (from `Simplex.Messaging.Agent.Store.Shared`) has fields `{name :: String, up :: Text, down :: Maybe Text}`. Initial migration has `Nothing` for `down`. Future migrations should include `Just down_migration` for rollback support. Called via `createDBStore dbOpts xftpServerMigrations (MigrationConfig confirmMigrations Nothing)`.
### Postgres Operations
Key query patterns:
- **`addFile`**: `INSERT INTO files (...) VALUES (...)`, return `DUPLICATE_` on unique violation.
- **`setFilePath`**: `UPDATE files SET file_path = ? WHERE sender_id = ? AND file_path IS NULL`, verified with `assertUpdated` (returns `AUTH` if 0 rows affected — file not found or already uploaded). The `WHERE file_path IS NULL` prevents duplicate uploads; the `UPDATE` acquires a row lock implicitly. Only persists the path; `usedStorage` managed by server.
- **`addRecipient`**: `INSERT INTO recipients (...)`, plus check for duplicates. No need for `recipientIds` TVar update — Postgres derives it from the table.
- **`getFile`** (sender): `SELECT ... FROM files WHERE sender_id = ?`, returns auth key from `sender_key` column.
- **`getFile`** (recipient): `SELECT f.*, r.recipient_key FROM recipients r JOIN files f ON ... WHERE r.recipient_id = ?`.
- **`deleteFile`**: `DELETE FROM files WHERE sender_id = ?` (recipients cascade).
- **`blockFile`**: `UPDATE files SET status = ? WHERE sender_id = ?`. When `deleted = True`, the server adjusts `usedStorage` externally (matching current STM behavior where `blockFile` only updates status and storage, not `filePath`).
- **`expiredFiles`**: `SELECT sender_id, file_path, file_size FROM files WHERE created_at + ? < ? LIMIT ?` — batched query replaces per-file iteration, includes `file_size` for `usedStorage` adjustment. Called in a loop until no rows returned.
The `iniFileContent` function in `Main.hs` must be updated to generate the new keys in the `[STORE_LOG]` section. Following SMP's `iniDbOpts` pattern with `optDisabled'` (prefixes `"# "` when value equals default), Postgres keys are generated commented out by default:
```ini
[STORE_LOG]
enable: on
# File storage mode: `memory` or `database` (PostgreSQL).
store_files: memory
# Database connection settings for PostgreSQL database (`store_files: database`).
Reuses `iniDBOptions` from `Simplex.Messaging.Server.CLI` for runtime parsing (falls back to defaults when keys are commented out or missing). `enableDbStoreLog'` pattern (`settingIsOn "STORE_LOG" "db_store_log"`) controls `dbStoreLogPath`.
No `--table` flag needed (unlike SMP which has queues/messages/all) — XFTP has a single entity type (files + recipients, always migrated together).
CLI options reuse `dbOptsP` parser from `Simplex.Messaging.Server.CLI`.
### Import (StoreLog -> PostgreSQL)
1. Confirm: prompt user with database connection details and StoreLog path
2. Read and replay StoreLog into temporary `STMFileStore`
3. Connect to PostgreSQL, run schema migrations (`createSchema = True`, `confirmMigrations = MCYesUp`)
4. Batch-insert file records into `files` table using PostgreSQL COPY protocol (matching SMP's `batchInsertQueues` pattern for performance). Progress reported every 10k files.
5. Batch-insert recipient records into `recipients` table using COPY protocol
6. Verify counts: `SELECT COUNT(*) FROM files` / `recipients` — warn if mismatch
7. Rename StoreLog to `.bak` (prevents accidental re-import, preserves original for rollback)
8. Report counts
### Export (PostgreSQL -> StoreLog)
1. Confirm: prompt user with database connection details and output path. Fail if output file already exists.
2. Connect to PostgreSQL
3. Open new StoreLog file for writing
4. Fold over all file records, writing per file (in this order, matching existing `writeFileStore`): `AddFile` (with `ServerEntityStatus` — this preserves `EntityBlocked` state), `AddRecipients`, then `PutFile` (if `file_path` is set)
5. Report counts
Note: `AddFile` carries `ServerEntityStatus` which includes `EntityBlocked info`, so blocking state is preserved through export/import without needing separate `BlockFile` log entries.
File data on disk is untouched by migration — only metadata moves between backends.
## Cabal Integration
Shared `server_postgres` flag. New Postgres modules added to existing conditional block:
- `Store.hs` — Postgres `FromField`/`ToField` instances for XFTP-specific types if needed
- `Env.hs` — `XSCDatabase` constructor
- `Main.hs` — database CLI commands, store selection for `database` mode, Postgres imports
- `Server.hs` — Postgres-specific imports if needed
## Testing
- **Parameterized server tests**: Existing `xftpServerTests` refactored to accept a store type parameter (following SMP's `SpecWith (ASrvTransport, AStoreType)` pattern). The same server tests run against both STM and Postgres backends — STM tests run unconditionally, Postgres tests added under `#if defined(dbServerPostgres)` with `postgressBracket` for database lifecycle (drop → create → test → drop).
- **Migration round-trip**: STM store → export to StoreLog → import to Postgres → export back → verify StoreLog equality (including blocked file status)
- **Tests location**: in `tests/` alongside existing XFTP tests, guarded by `server_postgres` CPP flag
- **Test database**: PostgreSQL on `localhost:5432`, using a dedicated `xftp_server_test` schema (dropped and recreated per test run via `postgressBracket`, following SMP's test database lifecycle pattern)
- **Test fixtures**: `testXFTPStoreDBOpts :: DBOpts` with `createSchema = True`, `confirmMigrations = MCYesUp`, in `tests/XFTPClient.hs`
> **For agentic workers:** REQUIRED: Use superpowers-extended-cc:subagent-driven-development (if subagents available) or superpowers-extended-cc:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add PostgreSQL backend support to xftp-server as an alternative to STM + StoreLog, with bidirectional migration.
**Architecture:** Introduce `FileStoreClass` typeclass (IO-based, following `QueueStoreClass` pattern). Extract current STM store into `Store/STM.hs`, make `Server.hs` polymorphic, then add `Store/Postgres.hs` behind `server_postgres` CPP flag. `usedStorage` moves from store to `XFTPEnv` so the server manages quota tracking externally.
**Tech Stack:** Haskell, postgresql-simple, STM, fourmolu, cabal with CPP flags
- [ ] **Step 1: Remove `usedStorage` from `FileStore` in `Store.hs`**
1. Remove `usedStorage :: TVar Int64` field from `FileStore` record (line 47).
2. Remove `usedStorage <- newTVarIO 0` from `newFileStore` (line 75) and drop the field from the record construction (line 76).
3. In `setFilePath` (line 92-97): remove `modifyTVar' (usedStorage st) (+ fromIntegral (size fileInfo))` — keep only `writeTVar filePath (Just fPath)`. Change pattern from `\FileRec {fileInfo, filePath}` to `\FileRec {filePath}` (fileInfo is now unused — `-Wunused-matches` error).
4. In `deleteFile` (line 112-119): remove `modifyTVar' usedStorage $ subtract (fromIntegral $ size fileInfo)`. Change outer pattern match from `FileStore {files, recipients, usedStorage}` to `FileStore {files, recipients}`. Change inner pattern from `Just FileRec {fileInfo, recipientIds}` to `Just FileRec {recipientIds}` (`fileInfo` is now unused — `-Wunused-matches` error).
5. In `blockFile` (line 122-127): remove `when deleted $ modifyTVar' usedStorage $ subtract (fromIntegral $ size fileInfo)`. Change pattern match from `st@FileStore {usedStorage}` to `st`. The `deleted` parameter and `fileInfo` in the inner pattern become unused — prefix with `_` or remove from pattern to avoid `-Wunused-matches`.
- [ ] **Step 2: Add `usedStorage` to `XFTPEnv` in `Env.hs`**
1. Add `usedStorage :: TVar Int64` field to `XFTPEnv` record (between `store` and `storeLog`, line 93).
2. In `newXFTPServerEnv` (line 112-126): replace lines 117-118:
```
used <- countUsedStorage <$> readTVarIO (files store)
5. Line 694: `FileStore {files, usedStorage} <- asks store` → split into `FileStore {files} <- asks store` and `usedStorage <- asks usedStorage`.
6. In `deleteOrBlockServerFile_` (line 620): after `void $ atomically $ storeAction st`, add usedStorage adjustment — `us <- asks usedStorage` then `atomically $ modifyTVar' us $ subtract (fromIntegral $ size fileInfo)` when file had a path (check `path` from `readTVarIO filePath` earlier in the function).
- [ ] **Step 4: Build and verify**
Run: `cabal build`
- [ ] **Step 5: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- [ ] **Step 1: Add three new functions to `Store.hs`**
1. Add to exports: `getUsedStorage`, `getFileCount`, `expiredFiles`.
2. Remove `expiredFilePath` from exports AND delete the function definition (dead code → `-Wunused-binds` error). Also remove `($>>=)` from import `Simplex.Messaging.Util (ifM, ($>>=))` → `Simplex.Messaging.Util (ifM)` — `$>>=` was only used by `expiredFilePath`.
3. Add import: `qualified Data.Map.Strict as M` (needed for `M.foldl'` in `getUsedStorage` and `M.toList` in `expiredFiles`).
5. Add imports: `Data.Maybe (catMaybes)`, `Data.Word (Word32)` (note: `qualified Data.Map.Strict as M` already added in item 3).
- [ ] **Step 2: Replace `countUsedStorage` in `Env.hs`**
1. Replace `countUsedStorage <$> readTVarIO (files store)` with `getUsedStorage store` in `newXFTPServerEnv`.
2. Remove `countUsedStorage` function definition and its export.
3. Remove `qualified Data.Map.Strict as M` import if no longer used.
- [ ] **Step 3: Update `restoreServerStats` in `Server.hs` to use `getFileCount`**
In `restoreServerStats` (line 694-696): replace `FileStore {files} <- asks store` and `_filesCount <- M.size <$> readTVarIO files` with `st <- asks store` and `_filesCount <- liftIO $ getFileCount st` (eliminates the `FileStore` pattern match — `files` binding no longer needed).
- [ ] **Step 4: Replace `expireServerFiles` iteration in `Server.hs`**
1. Replace the body of `expireServerFiles` (lines 636-660). Remove `files' <- readTVarIO (files st)` and the `forM_ (M.keys files')` loop.
2. New body: call `expiredFiles st old 10000` in a loop. For each `(sId, filePath_, fileSize)` in returned list: apply `itemDelay`, remove disk file if present, call `atomically $ deleteFile st sId`, adjust `usedStorage` TVar by `fileSize`, increment `filesExpired` stat. Loop until `expiredFiles` returns `[]`.
3. Remove `Data.Map.Strict` import from Server.hs if no longer needed (was used for `M.size` and `M.keys` — now replaced by `getFileCount` and `expiredFiles`).
- [ ] **Step 5: Build and verify**
Run: `cabal build`
- [ ] **Step 6: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
1. Line 563 (`receiveServerFile`): change `atomically $ writeTVar filePath (Just fPath)` → add `st <- asks store` then `void $ liftIO $ setFilePath st senderId fPath` (design call site #1 — `store` is not in scope in `receiveServerFile`'s `receive` helper, so bind via `asks`; `void` avoids `-Wunused-do-bind` warning on the `Either` result).
2. Line 453 (`verifyXFTPTransmission`): split `atomically $ verify =<< getFile st party fId` into: `liftIO (getFile st party fId)` (IO→M lift), then pattern match on result, use `readTVarIO (fileStatus fr)` instead of `readTVar`.
3. Lines 371, 377 (control port `CPDelete`/`CPBlock`): change `ExceptT $ atomically $ getFile fs SFRecipient fileId` → `ExceptT $ liftIO $ getFile fs SFRecipient fileId` (inside `unliftIO u $ do` block which runs in M monad — `liftIO` required to lift IO into M).
4. Line 508 (`addFile` in `createFile`): the `ExceptT $ addFile st sId file ts EntityActive` — `addFile` is now IO, `ExceptT` wraps IO directly. Remove any `atomically`.
5. Line 514 (`addRecipient`): same — `ExceptT . addRecipient st sId` works directly in IO.
6. Line 516 (`retryAdd`): change parameter type from `(XFTPFileId -> STM (Either XFTPErrorType a))` to `(XFTPFileId -> IO (Either XFTPErrorType a))`. Line 520: change `atomically (add fId)` to `liftIO (add fId)`.
7. Line 605 (`ackFileReception`): change `atomically $ deleteRecipient st rId fr` to `liftIO $ deleteRecipient st rId fr`.
8. Line 620 (`deleteOrBlockServerFile_`): change third parameter type from `(FileStore -> STM (Either XFTPErrorType ()))` to `(FileStore -> IO (Either XFTPErrorType ()))`. Line 626: change `void $ atomically $ storeAction st` to `void $ liftIO $ storeAction st`.
9. `expireServerFiles``delete` helper: change `atomically $ deleteFile st sId` to `liftIO $ deleteFile st sId` (deleteFile is now IO; `liftIO` required because the helper runs in M monad, not IO).
3. Move from `Store.hs`: `FileStore` data type (rename to `STMFileStore`), all function implementations, internal helpers (`withFile`, `newFileRec`), all STM-specific imports.
4. Rename all `FileStore` references to `STMFileStore` in the new file.
5. Module declaration: `module Simplex.FileTransfer.Server.Store.STM` exporting only `STMFileStore (..)` — do NOT export standalone functions (`addFile`, `setFilePath`, etc.) to avoid name collisions with the typeclass methods from `Store.hs`.
- [ ] **Step 2: Rewrite `Store.hs` as the typeclass module**
1. Add `{-# LANGUAGE TypeFamilies #-}` pragma to `Store.hs` (required for `type FileStoreConfig s` associated type).
2. Keep in `Store.hs`: `FileRec (..)`, `FileRecipient (..)`, `RoundedFileTime`, `fileTimePrecision` definitions and their `StrEncoding` instance.
deleteRecipient :: s -> RecipientId -> FileRec -> IO ()
ackFile :: s -> RecipientId -> IO (Either XFTPErrorType ())
-- Expiration
expiredFiles :: s -> Int64 -> Int -> IO [(SenderId, Maybe FilePath, Word32)]
-- Stats
getUsedStorage :: s -> IO Int64
getFileCount :: s -> IO Int
```
4. Do NOT re-export from `Store/STM.hs` — this would create a circular module dependency (Store.hs imports Store/STM.hs, Store/STM.hs imports Store.hs). Consumers must import `Store.STM` directly where they need `STMFileStore`.
5. Remove all STM-specific imports that are no longer needed.
- [ ] **Step 3: Add `FileStoreClass` instance in `Store/STM.hs`**
1. Import `FileStoreClass` from `Simplex.FileTransfer.Server.Store`.
2. Inline all implementations directly in the instance body (do NOT delegate to standalone functions — the standalone names collide with typeclass method names, causing ambiguous occurrences for importers):
```haskell
instance FileStoreClass STMFileStore where
type FileStoreConfig STMFileStore = ()
newFileStore () = do
files <- TM.emptyIO
recipients <- TM.emptyIO
pure STMFileStore {files, recipients}
closeFileStore _ = pure ()
addFile st sId fileInfo createdAt status = atomically $ ...
setFilePath st sId fPath = atomically $ ...
-- ... (each method's body is the existing function body, inlined)
```
3. Remove the standalone top-level function definitions — they are now instance methods. Keep only `withFile` and `newFileRec` as internal helpers used by the instance methods.
- [ ] **Step 4: Update importers**
1. `Env.hs`: add `import Simplex.FileTransfer.Server.Store.STM (STMFileStore (..))`. Change `FileStore` → `STMFileStore` in `XFTPEnv` type and `newXFTPServerEnv`. Change `store <- newFileStore` to `store <- newFileStore ()` (typeclass method now takes `FileStoreConfig STMFileStore` which is `()`). Keep `import Simplex.FileTransfer.Server.Store` for `FileRec`, `FileRecipient`, `FileStoreClass`, etc.
2. `Server.hs`: add `import Simplex.FileTransfer.Server.Store.STM`. Change `FileStore` → `STMFileStore` in any explicit type annotations. Import `FileStoreClass` from `Simplex.FileTransfer.Server.Store`.
3. `StoreLog.hs`: add `import Simplex.FileTransfer.Server.Store.STM` to access concrete `STMFileStore` type and store functions used during log replay. Change `FileStore` → `STMFileStore` in `readWriteFileStore` and `writeFileStore` parameter types.
- [ ] **Step 5: Update cabal file**
Add `Simplex.FileTransfer.Server.Store.STM` to `exposed-modules` in the `!flag(client_library)` section, alongside existing XFTP server modules.
- [ ] **Step 6: Build and verify**
Run: `cabal build`
- [ ] **Step 7: Run existing tests**
Run: `cabal test --test-show-details=streaming --test-option=--match="/XFTP/"`
- Modify: `tests/XFTPClient.hs` (if it calls `runXFTPServerBlocking` directly)
- [ ] **Step 1: Make `XFTPEnv` polymorphic in `Env.hs`**
1. Add `XFTPStoreConfig` GADT: `data XFTPStoreConfig s where XSCMemory :: Maybe FilePath -> XFTPStoreConfig STMFileStore`.
2. Change `data XFTPEnv` to `data XFTPEnv s` — field `store :: FileStore` becomes `store :: s`.
3. Change `newXFTPServerEnv :: XFTPServerConfig -> IO XFTPEnv` to `newXFTPServerEnv :: FileStoreClass s => XFTPStoreConfig s -> XFTPServerConfig -> IO (XFTPEnv s)`.
4. Pattern match on `XSCMemory storeLogPath` in `newXFTPServerEnv` body. Create store via `newFileStore ()`, storeLog via `mapM (`readWriteFileStore` st) storeLogPath`.
- [ ] **Step 2: Make `Server.hs` polymorphic**
1. Change `type M a = ReaderT XFTPEnv IO a` to `type M s a = ReaderT (XFTPEnv s) IO a`.
2. Add `FileStoreClass s =>` constraint to all functions using `M s a`. Use `forall s.` in signatures of functions that have `where`-block bindings with `M s` type annotations — `ScopedTypeVariables` requires explicit `forall` to bring `s` into scope for inner type signatures (matching SMP's `smpServer :: forall s. MsgStoreClass s => ...` pattern). Full list: `xftpServer`, `processRequest`, `verifyXFTPTransmission`, `processXFTPRequest` and all its `where`-bound functions (`createFile`, `addRecipients`, `receiveServerFile`, `sendServerFile`, `deleteServerFile`, `ackFileReception`, `retryAdd`, `addFileRetry`, `addRecipientRetry`), `deleteServerFile_`, `blockServerFile`, `deleteOrBlockServerFile_`, `expireServerFiles`, `randomId`, `getFileId`, `withFileLog`, `incFileStat`, `saveServerStats`, `restoreServerStats`, `randomDelay` (inside `#ifdef slow_servers` CPP block). Also update `encodeXftp` (line 236) and `runCPClient` (line 339) which use explicit `ReaderT XFTPEnv IO` instead of the `M` alias — change to `ReaderT (XFTPEnv s) IO`.
3. Change `runXFTPServerBlocking` and `runXFTPServer` to take `XFTPStoreConfig s` parameter.
4. Add `closeFileStore store` call to the server shutdown path (in the `finally` block or `stopServer` equivalent — after saving stats, before logging "Server stopped"). This ensures Postgres connection pool and `dbStoreLog` are properly closed. For STM this is a no-op.
- [ ] **Step 3: Update `Main.hs` dispatch**
1. In `runServer`: construct `XSCMemory (enableStoreLog $> storeLogFilePath)`.
2. Add dispatch function that calls the updated `runXFTPServer` (which creates `started` internally):
```haskell
run :: FileStoreClass s => XFTPStoreConfig s -> IO ()
run storeCfg = runXFTPServer storeCfg serverConfig
```
3. Call `run` with the `XSCMemory` config.
- [ ] **Step 4: Update test helper if needed**
If `tests/XFTPClient.hs` calls `runXFTPServerBlocking` directly, update the call to pass an `XSCMemory` config. Check the `withXFTPServer` / `serverBracket` helper.
Full migration module with `xftpServerMigrations :: [Migration]` and `m20260325_initial` containing CREATE TABLE SQL for `files` and `recipients` tables plus indexes. Follow SMP's `QueueStore/Postgres/Migrations.hs` pattern exactly: tuple list → `sortOn name . map migration`.
- [ ] **Step 3: Create `Store/Postgres.hs` with stub instance**
1. Define `PostgresFileStore` with `dbStore :: DBStore` and `dbStoreLog :: Maybe (StoreLog 'WriteMode)`.
2. `instance FileStoreClass PostgresFileStore` with `error "not implemented"` for all methods except `newFileStore` (calls `createDBStore` + opens `dbStoreLog`) and `closeFileStore` (closes both). `type FileStoreConfig PostgresFileStore = PostgresFileStoreCfg`.
For `RoundedFileTime` (Int64 wrapper), `ServerEntityStatus` (Text via StrEncoding), `C.APublicAuthKey` (Binary via `encodePubKey`/`decodePubKey`). Check SMP's `QueueStore/Postgres.hs` for existing instances to import.
- [ ] **Step 9: Wrap mutation operations in `uninterruptibleMask_`**
Operations that combine a DB write with a TVar update (e.g., `getFile` constructs `FileRec` with `newTVarIO`) must be wrapped in `E.uninterruptibleMask_` to prevent async exceptions from leaving inconsistent state. Follow SMP's `addQueue_`, `deleteStoreQueue` pattern.
- [ ] **Step 1: Update `iniFileContent` in `Main.hs`**
Add to `[STORE_LOG]` section: `store_files: memory`, commented-out `db_connection`, `db_schema`, `db_pool_size`, `db_store_log` keys. Follow SMP's `optDisabled'` pattern for commented defaults.
- [ ] **Step 2: Add `StartOptions` and `--confirm-migrations` flag**
```haskell
data StartOptions = StartOptions
{ confirmMigrations :: MigrationConfirmation
}
```
Add to `Start` command parser with default `MCConsole`. Thread through to `runServer`.
- [ ] **Step 3: Add store_files INI parsing and CPP-guarded Postgres dispatch**
In `runServer`: read `store_files` from INI (`fromRight "memory" $ lookupValue "STORE_LOG" "store_files" ini`). Add `"database"` branch (CPP-guarded) that constructs `PostgresFileStoreCfg` using `iniDBOptions ini defaultXFTPDBOpts` and `enableDbStoreLog'` pattern. Non-postgres build: `exitError`.
Add `testXFTPDBConnectInfo :: ConnectInfo` matching the connection string.
- [ ] **Step 2: Add Postgres server test group in `tests/Test.hs`**
CPP-guarded block that runs existing `xftpServerTests` with Postgres store config, wrapped in `postgressBracket testXFTPDBConnectInfo`. Parameterize `withXFTPServer` to accept store config if needed.
- [ ] **Step 3: Create `tests/CoreTests/XFTPStoreTests.hs` — unit tests**
Test `PostgresFileStore` operations directly:
- `addFile` + `getFile SFSender` round-trip.
- `addFile` duplicate → `DUPLICATE_`.
- `getFile` nonexistent → `AUTH`.
- `setFilePath` + verify `WHERE file_path IS NULL` guard.
Create `STMFileStore` with test data (files + recipients + blocked status) → export to StoreLog → import to Postgres → export back → compare StoreLog files byte-for-byte.
- [ ] **Step 5: Build and run tests**
```bash
cabal build -fserver_postgres test:simplexmq-test
cabal test --test-show-details=streaming --test-option=--match="/XFTP/" -fserver_postgres
Haskell FFI bindings to libbbs for BBS+ signatures. General-purpose - the module knows nothing about specific applications.
## How BBS+ works
BBS+ signs a fixed list of N messages. Each message is an arbitrary byte array. The signer signs all N messages at once with one signature.
The holder of the signature can then generate a proof that selectively discloses some messages and hides others. The verifier learns the disclosed messages and confirms they were signed by the signer, but learns nothing about the hidden messages. Different proofs from the same signature are unlinkable.
Key constraint: the total number of messages N is fixed at signing time. The verifier must know N. A proof generated from a 3-message signature cannot be verified as a 2-message proof.
The recipient only sees the proof, presentationHeader, expiry string, and badge type string. They verify these were signed by the server (pk is hardcoded). They never see the master secret.
Expiry is always present as a string. Monthly badges use a date like `"2026-07-31"`, lifetime badges use `"lifetime"`. BBS+ doesn't interpret the bytes - expiry semantics are the application's responsibility. This keeps the message count fixed at 3 for all badge types.
## libbbs C API mapping
```c
int bbs_keygen_full(ciphersuite, sk, pk)
int bbs_sign(ciphersuite, sk, pk, signature, header, header_len, n, messages, message_lens)
int bbs_proof_gen(ciphersuite, pk, signature, proof, header, header_len, presentation_header, presentation_header_len, disclosed_indexes, disclosed_indexes_len, n, messages, message_lens)
int bbs_proof_verify(ciphersuite, pk, proof, proof_len, header, header_len, presentation_header, presentation_header_len, disclosed_indexes, disclosed_indexes_len, n, messages, message_lens)
```
We use `bbs_sha256_ciphersuite`. The header parameter is exposed in all Haskell functions - the application decides what to put there. Tests use `"SimpleX"` as header.
The `presentation_header` parameter is what we call `presentationHeader`.
In `bbs_proof_verify`, the `n` parameter is the total number of messages (not the number of disclosed messages). The `messages` array contains only the disclosed messages, and `disclosed_indexes` maps each to its position in the original message list.
- [Connection link: 1-time invitation and contact address](#connection-link-1-time-invitation-and-contact-address)
- [Appendix A: SMP agent API](#smp-agent-api)
- [API functions](#api-functions)
- [API events](#api-events)
## Abstract
The purpose of SMP agent protocol is to define the syntax and the semantics of communications between the client and the agent that connects to [SMP](./simplex-messaging.md) servers.
It provides:
- protocol to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the servers location from the users of the agent protocol.
- API to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the servers location from the users of the agent protocol.
- management of E2E encryption between SMP agents, generating ephemeral asymmetric keys for each connection.
- SMP command authentication on SMP servers, generating ephemeral keys for each SMP queue.
- TCP/TLS transport handshake with SMP servers.
- validation of message integrity.
SMP agent protocol provides no encryption or security on the client side - it is assumed that the agent is executed in the trusted and secure environment, in one of three ways:
- via TCP network using secure connection.
- via local port (when the agent runs on the same device as a separate process).
- via agent library, when the agent logic is included directly into the client application - [SimpleX Chat for terminal](https://github.com/simplex-chat/simplex-chat) uses this approach.
SMP agent API provides no security between the agent and the client - it is assumed that the agent is executed in the trusted and secure environment, via the agent library, when the agent logic is included directly into the client application - [SimpleX Chat for terminal](https://github.com/simplex-chat/simplex-chat) uses this approach.
## SMP agent
SMP agents communicate with each other via SMP servers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the commands received from its users. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections").
SMP agents communicate with each other via SMP servers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the API calls used by the client applications. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections").
The agent must have a persistent storage to manage the states of known connections and of the client-side information of SMP queues that each connection consists of, and also the buffer of the most recent sent and received messages. The number of the messages that should be stored is implementation specific, depending on the error management approach that the agent implements; at the very least the agent must store the hashes and IDs of the last received and sent messages.
## SMP servers management
SMP agent protocol commands do not contain the addresses of the SMP servers that the agent will use to create and use the connections (excluding the server address in queue URIs used in JOIN command). The list of the servers is a part of the agent configuration and can be dynamically changed by the agent implementation:
SMP agent API does not use the addresses of the SMP servers that the agent will use to create and use the connections (excluding the server address in queue URIs used in JOIN command). The list of the servers is a part of the agent configuration and can be dynamically changed by the agent implementation:
- by the client applications via any API that is outside of scope of this protocol.
- by the agents themselves based on availability and latency of the configured servers.
## SMP agent protocol components
## SMP agent protocol scope
SMP agent protocol has 3 main parts:
SMP agent protocol has 2 main parts:
- the syntax and semantics of the messages that SMP agents exchange with each other in order to:
- the messages that SMP agents exchange with each other in order to:
- negotiate establishing unidirectional (simplex) encrypted queues on SMP servers.
- exchange client messages and delivery notifications, providing sequential message IDs and message integrity (by including the hash of the previous message).
- the syntax and semantics of the commands that are sent by the agent clients to the agents. This protocol allows to create and manage multiple connections, each consisting of two or more SMP queues.
- the syntax and semantics of the message that the clients of SMP agents should send out-of-band (as pre-shared "invitation" including queue URIs) to protect [E2E encryption][1] from active attacks ([MITM attacks][2]).
- re-negotiate messaging queues to use and connection e2e encryption.
- the messages that the clients of SMP agents should send out-of-band (as pre-shared "invitation" including queue URIs) to protect [E2E encryption][1] from active attacks ([MITM attacks][2]).
[Appendix A](#appendix-a-smp-agent-api) of this document describes:
- the functional API used by the client application with the agent. This API allows to create and manage multiple connections, each consisting of two or more SMP queues.
- events that the agent passes to the clients.
## Duplex connection procedure
@@ -75,54 +69,126 @@ SMP agent protocol has 3 main parts:
The procedure of establishing a duplex connection is explained on the example of Alice and Bob creating a bi-directional connection consisting of two unidirectional (simplex) queues, using SMP agents (A and B) to facilitate it, and two different SMP servers (which could be the same server). It is shown on the diagram above and has these steps:
1. Alice requests the new connection from the SMP agent A using SMP NEW command.
2. Agent A creates an SMP connection on the server (using [SMP protocol](./simplex-messaging.md)) and responds to Alice with the invitation that contains queue information and the encryption key Bob's agent B should use. The invitation format is described in [Connection request](#connection-request).
3. Alice sends the [connection request](#connection-request) to Bob via any secure channel (out-of-band message).
4. Bob sends `JOIN` command with the connection request as a parameter to agent B to accept the connection.
- Agent B sends an "SMP confirmation" with SMP SEND command to the SMP queue specified in the connection request - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, etc.). This message is encrypted using key passed in the connection request (or with the derived key, in which case public key for key derivation should be sent in clear text).
- Agent A receives the SMP confirmation containing Bob's key and info as SMP MSG.
- Agent A notifies Alice sending REQ notification with Bob's info.
- Alice accepts connection request with ACPT command.
- Agent A secures the queue with SMP KEY command.
- Agent B tries sending authenticated SMP SEND command with agent `HELLO` message until it succeeds. Once it succeeds, Bob's agent "knows" the queue is secured.
6. Agent B creates a new SMP queue on the server.
7. Establish Bob's SMP queue:
- Agent B sends `REPLY` message (SMP SEND command) with the connection request to this 2nd queue to Alice's agent (via the 1st queue) - this connection request SHOULD use "simplex" URI scheme.
- Agent A, having received `REPLY` message, sends unauthenticated message (SMP SEND) to SMP queue with Alice agent's ephemeral key that will be used to authenticate Alice's commands to the queue, as described in SMP protocol, and Alice's info.
- Bob's agent receives the key and Alice's information and secures the queue (SMP KEY).
- Bob's agent sends the notification `INFO` with Alice's information to Bob.
- Alice's agent keeps sending `HELLO` message until it succeeds.
8. Agents A and B notify Alice and Bob that connection is established.
- Once sending`HELLO`succeeds, Alice's agent sends to Alice `CON` notification that confirms that now both parties can communicate.
- Once Bob's agent receives `HELLO` from Alice's agent, it sends to Bob `CON` notification as well.
1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function.
2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md)`NEW` command) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
3. Alice sends the [connection link](#connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code.
4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection.
5. Agent B creates Bob's SMP reply queue with SMP server `NEW` command.
6. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
6. Alice confirms and continues the connection:
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`.
- Agent A notifies Alice sending `CONF` notification with Bob's info.
- Alice allows connection to continue with agent `allowConnection` api function.
- Agent A secures the queue with SMP server `KEY` command.
- Agent A sends SMP confirmation with ephemeral sender key, ephemeral public encryption key and profile (but without reply queue).
7. Agent B confirms the connection:
- receives the confirmation.
- sends the notification `INFO` with Alice's information to Bob.
- secures SMP queue that it sent to Alice in the first confirmation with SMP `KEY` command .
- sends `HELLO` message via SMP `SEND` command. This confirms that the reply queue is secured and also validates that Agent A secured the first SMP queue
8. Agent A notifies Alice.
- receives `HELLO` message from Agent B.
- sends`HELLO`message to Agent B via SMP `SEND` command.
- sends `CON` notification to Alice, confirming that the connection is established.
9. Agent B notifies Bob.
- Once Agent B receives `HELLO` from Agent A, it sends to Bob `CON` notification as well.
At this point the duplex connection between Alice and Bob is established, they can use `SEND` command to send messages. The diagram also shows how the connection status changes for both parties, where the first part is the status of the SMP queue to receive messages, and the second part - the status of the queue to send messages.
The most communication happens between the agents and servers, from the point of view of Alice and Bob there are 4 steps (not including notifications):
1. Alice requests a new connection with `NEW` command and receives the invitation.
2. Alice passes connection request out-of-band to Bob.
3. Bob accepts the connection with `JOIN` command with the connection request to his agent.
4. Alice accepts the connection with `ACPT`command.
1. Alice requests a new connection with `createConnection` agent API function and receives the connection link.
2. Alice passes connection link out-of-band to Bob.
3. Bob accepts the connection with `joinConnection` agent API function with the connection link to his agent.
4. Alice accepts the connection with `ACPT`agent API function.
5. Both parties receive `CON` notification once duplex connection is established.
Clients SHOULD support establishing duplex connection asynchronously (when parties are intermittently offline) by persisting intermediate states and resuming SMP queue subscriptions.
## Fast duplex connection procedure
Previously described duplex connection procedure requires sending 4 messages creating a bad UX for the users - it requires waiting until each party in online before the messages can be sent.
It allows users validating connecting party profile before proceeding with the connection, but it turned out to be unnecessary UX step and is not used in the client applications.
It also protects against an attacker who compromised TLS and uses the sender queue ID sent to the recipient to secure the queue before the sender can. This attack is very hard, and this accepting its risk is better than worse UX. Future protocol versions could mitigate this attack by encrypting entity IDs.
Faster duplex connection process is possible with the `SKEY` command added in v9 of SMP protocol.
1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function
2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md) `NEW` command with the flag allowing the sender to secure the queue) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address).
3. Alice sends the [connection link](connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code. This link contains the flag that the queue can be secured by the sender.
4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection.
5. Agent B secures Alice's queue with SMP command `SKEY` - this command can be proxied.
6. Agent B creates Bob's SMP reply queue with SMP server `NEW` command (with the flag allowing the sender to secure the queue).
7. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text).
8. Alice confirms the connection:
- Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`.
- Agent A notifies Alice sending `CONF` notification with Bob's info (that indicates that Agent B already secured the queue).
- Alice allows connection to continue with agent `allowConnection` api function.
- Agent A secures Bob's queue with SMP command `SKEY`.
- Agent A sends SMP confirmation with ephemeral public encryption key and profile (but without reply queue, and without sender key).
9. Agent A notifies Alice with `CON` notification.
10. Agent B notifies Bob about connection success:
- receives confirmation message from Alice.
- sends the notification `INFO` with Alice's information to Bob.
- sends `CON` notification to Bob.
## Contact addresses
SMP agents support creating a special type of connection - a contact address - that allows to connect to multiple network users who can send connection requests by sending 1-time connection links to the message queue.
This connection address uses a messaging queue on SMP server to receive invitations to connect - see `agentInvitation` message below. Once connection request is accepted, a new connection is created and the address itself is no longer used to send the messages - deleting this address does not disrupt the connections that were created via it.
## Communication between SMP agents
To establish duplex connections and to send messages on behalf of their clients, SMP agents communicate via SMP servers.
Agents use SMP message client body (the part of the SMP message after header - see [SMP protocol](./simplex-messaging.md)) to transmit agent client messages and exchange messages between each other.
Each SMP message client body, once decrypted, contains 3 parts (one of them may include binary message body), as defined by `decryptedSmpMessageBody` syntax:
These messages are encrypted with per-queue shared secret using NaCL crypto_box and can be of 4 types, as defined by `decryptedSMPClientMessage`:
- `agentConfirmation` - used when confirming SMP queues, contains connection information encrypted with double ratchet. This envelope can only contain `agentConnInfo` or `agentConnInfoReply` encrypted with double ratchet.
- `agentMsgEnvelope` - contains different agent messages encrypted with double ratchet, as defined in `agentMessage`.
- `agentInvitation` - sent to SMP queue that is used as contact address, does not use double ratchet.
- `agentRatchetKey` - used to re-negotiate double ratchet encryption - can contain additional information in `agentRatchetKey`.
This syntax of decrypted SMP client message body is defined by `decryptedAgentMessage` below.
Decrypted SMP message client body can be one of 4 types:
- `agentConnInfo` - used by the initiating party when confirming reply queue - sent in `agentConfirmation` envelope.
- `agentConnInfoReply` - used by accepting party, includes reply queue(s) in the initial confirmation - sent in `agentConfirmation` envelope.
- `agentRatchetInfo` - used to pass additional information when renegotiating double ratchet encryption - sent in `agentRatchetKey` envelope.
- `agentMessage` - all other agent messages.
`agentMessage` contains these parts:
- `agentMsgHeader` - agent message header that contains sequential agent message ID for a particular SMP queue, agent timestamp (ISO8601) and the hash of the previous message.
- `agentMessage` - a command/message to the other SMP agent:
- to establish the connection with two SMP queues (`helloMsg`, `replyQueueMsg`)
- to send and to acknowledge user messages (`clientMsg`, `acknowledgeMsg`)
- to manage SMP queue rotation (`newQueueMessage`, `deleteQueueMsg`)
- to manage encryption key rotation (TODO)
- `aMessage` - a command/message to the other SMP agent:
- to confirm the connection (`HELLO`).
- to send and to confirm reception of user messages (`A_MSG`, `A_RCVD`).
- to confirm that the new double ratchet encryption is agreed (`EREADY`).
- to notify another party that it can continue sending messages after queue capacity was exceeded (`A_QCONT`).
- to manage SMP queue rotation (`QADD`, `QKEY`, `QUSE`, `QTEST`).
- `msgPadding` - an optional message padding to make all SMP messages have constant size, to prevent servers from observing the actual message size. The only case the message padding can be absent is when the message has exactly the maximum size, in all other cases the message MUST be padded to a fixed size.
### Messages between SMP agents
@@ -130,269 +196,160 @@ Each SMP message client body, once decrypted, contains 3 parts (one of them may
Message syntax below uses [ABNF][3] with [case-sensitive strings extension][4].
newQueueInfo = version smpServer senderId dhPublicKey [sndSecure]
senderKey = length x509encoded
QUSE = %s"QU" sndQueuesReady
sndQueuesReady = length 1*(sndQueueAddr primary)
primary = %s"T" / %s"F"
QTEST = %s"QT" sndQueueAddrs
sndQueueAddrs = length 1*sndQueueAddr
sndQueueAddr = smpServer senderId
smpServer = hosts port keyHash
hosts = length 1*host
host = shortString
port = shortString
keyHash = shortString
senderId = shortString
clientVRange = version version
version = 2*2 OCTET
msgPadding = *OCTET
rcptLength = 2*2 OCTET
shortString = length *OCTET
length = 1*1 OCTET
```
#### HELLO message
This is the first message that both agents send after the respective SMP queue is secured by the receiving agent (see diagram). It MAY contain the public key that the recipient would use to verify messages signed by the sender.
This is the first message that both agents send after the respective SMP queue is secured by the receiving agent (see diagram).
Sending agent might need to retry sending HELLO message, as it would not have any other confirmation that the queue is secured other than the success of sending this message with the signed SMP SEND command.
This message is not used with [fast duplex connection](#fast-duplex-connection-procedure).
#### REPLY message
#### A_MSG message
This is the message that is sent by the agent that received an out-of-band connection request to pass the connection request for the reply SMP queues to the agent that originated the connection (see diagram).
This is the agent envelope used to send client messages once the connection is established. This is different from the MSG sent by SMP server to the agent and MSG event from SMP agent to the client that are sent in different contexts.
#### MSG message
#### A_RCVD message
This is the agent envelope used to send client messages once the connection is established. Do not confuse it with the MSG response from SMP server to the agent and MSG response from SMP agent to the client that are sent in different contexts.
This message is sent to confirm the client message reception. It includes received message number and message hash.
#### INV message
#### EREADY message
This message is sent to the SMP queue(s) in `connReqContact`, to establish a new connection via existing unsecured queue, that acts as a permanent connection link of a user.
This message is sent after re-negotiating a new double ratchet encryption with `agentRatchetKey`.
#### ACK message
#### A_QCONT message
This message is sent to confirm the client message reception. It includes received message number, message hash and the reception status.
This message is sent to notify the sender client that it can continue sending the messages after queue capacity was exhausted.
#### NEW message
### Rotating messaging queue
This message is sent to add an additional SMP queue to the connection. Unlike REPLY message it can be sent at any time.
SMP agents SHOULD support 4 messages to rotate message reception to another messaging server:
`QADD`: add the new queue address(es) to the connection - sent by the client that initiates rotation.
`QKEY`: pass sender's key via existing connection (SMP confirmation message will not be used, to avoid the same "race" of the initial key exchange that would create the risk of intercepting the queue for the attacker) - sent by the client accepting the rotation
`QUSE`: instruct the sender to use the new queue with sender's queue ID as parameter. From this point some messages can be sent to both the new queue and the old queue.
`QTEST`: send test message to the new connection. Any other message can be sent if available to continue rotation, the absence of this message is not an error. Once this message is successfully sent the sender will stop using the old queue. Once this message (or any other message in the new queue) is received, the recipient will stop using the old queue and delete it.
#### DEL message
**Queue rotation procedure**
This message is sent to notify that the queue with passed URI will be deleted - having received this message, the receiving agent should no longer send messages to this queue. In case it was the last remaining send queue in the duplex connection, the agent MAY also delete the reply queue(s) in the connection.
`SKEY` command added in v9 of SMP protocol allows for faster queue rotation procedure.
This part describes the transmissions between users and client-side SMP agents: commands that the users send to create and operate duplex connections and SMP agent responses and messages they deliver.
**Fast queue rotation procedure**
Commands syntax below is provided using [ABNF][3] with [case-sensitive strings extension][4].
| confirmation, in reply to: <br>no-pq inv <br>pq inv | <br>+<br>+ | <br>+<br>- | <br>-<br>+ |
| 1st msg, in reply to: <br>no-pq conf <br>pq/pq+ct conf | <br>+<br>+ | <br>+<br>- | <br>-<br>+ |
| Nth msg, in reply to: <br>no-pq msg <br>pq/pq+ct msg | <br>+<br>+ | <br>+<br>- | <br>-<br>+ |
invitation = %s"INV" SP connectionRequest ; `connectionRequest` is defined below
These scenarios can be reduced to:
1. initial invitation optionally has PQ key, but must not have ciphertext.
2. all subsequent messages should be allowed without PQ key/ciphertext, but:
- if the previous message had PQ key or PQ key with ciphertext, they must either have no PQ key, or have PQ key with ciphertext (PQ key without ciphertext is an error).
- if the previous message had no PQ key, they must either have no PQ key, or have PQ key without ciphertext (PQ key with ciphertext is an error).
confMsg = %s"CONF" SP confirmationId SP msgBody
; msgBody here is any binary information identifying connection request
The rules for calculating the shared secret for received/sent messages are (assuming received message is valid according to the above rules):
letCmd = %s"LET" SP confirmationId SP msgBody
; msgBody here is any binary information identifying connecting party
| sent msg > <br>V received msg | no-pq | pq | pq+ct |
| pq (sent msg was NOT pq) | DH / DH | err | DH / DH+KEM |
| pq+ct (sent msg was NOT no-pq) | DH+KEM / DH | err | DH+KEM / DH+KEM |
confirmationId = 1*DIGIT
To summarize, the upgrade to DH+KEM secret happens in a sent message that has PQ key with ciphertext sent in reply to message with PQ key only (without ciphertext), and the downgrade to DH secret happens in the message that has no PQ key.
connReqMsg = %s"REQ" SP invitationId SP msgBody
; msgBody here is any binary information identifying connection request
## Connection link: 1-time invitation and contact address
acceptCmd = %s"ACPT" SP invitationId SP msgBody
; msgBody here is any binary information identifying connecting party
Connection links are generated by SMP agent in response to `createConnection` api call, used by another party user with `joinConnection` api, and then another connection link is sent by the agent in `agentConnInfoReply` and used by the first party agent to connect to the reply queue (the second part of the process is invisible to the users).
invitationId = 1*DIGIT
connInfo = %s"INFO" SP msgBody
; msgBody here is any binary information identifying connecting party
connected = %s"CON"
subscribeCmd = %s"SUB" ; response is `ok` or `error`
unsubscribed = %s"END"
; when another agent (or another client of the same agent)
; subscribes to the same SMP queue on the server
connDown = %s"DOWN"
; lost connection (e.g. because of Internet connectivity or server is down)
badMsgIdErr = %s"ID" SP previousMsgId ; ID is lower than the previous
badHashErr = %s"HASH"
missingFromMsgId = agentMsgId
missingToMsgId = agentMsgId
previousMsgId = agentMsgId
acknowledgeCmd = %s"ACK" SP agentMsgId ; ID assigned by receiving agent (in MSG "R")
received = %s"RCVD" SP agentMsgId SP msgIntegrity
; ID assigned by sending agent (in SENT response)
; currently not implemented
msgStatus = ok | error
ok = %s"OK"
error = %s"ERR" SP <errorType>
```
### Client commands and server responses
#### NEW command and INV response
`NEW` command is used to create a connection and a connection request to be sent out-of-band to another protocol user (the joining party). It should be used by the client of the agent that initiates creating a duplex connection (the initiating party).
`INV` response is sent by the agent to the client of the initiating party.
`NEW` command has `connectionMode` parameter to define the connection mode - to be used to communicate with a single contact (invitation mode, `connectionMode` is `INV`) or to accept connection requests from anybody (contact mode, `connectionMode` is `CON`). The type of connection request is determined by `connectionMode` parameter.
#### JOIN command
It is used to create a connection and accept the connection request received out-of-band. It should be used by the client of the agent that accepts the connection (the joining party).
#### CONF notification and LET command
When the joining party uses `JOIN` command to accept connection invitation created with `NEW INV` command, the initiating party will receive `CONF` notification with some numeric identifier and an additional binary information, that can be used to identify the joining party or for any other purpose.
To continue with the connection the initiating party should use `LET` command.
#### REQ notification and ACPT command
When the joining party uses `JOIN` command to connect to the contact created with `NEW CON` command, the initiating party will receive `REQ` notification with some numeric identifier and an additional binary information, that can be used to identify the joining party or for any other purpose.
To continue with the connection the party that created the contact should use `ACPT` command.
#### INFO and CON notifications
After the initiating party proceeds with the connection using `ACPT` command, the joining party will receive `INFO` notification that can be used to identify the initiating party or for any other purpose.
Once the connection is established and ready to accept client messages, both agents will send `CON` notification to their clients.
#### SUB command
This command can be used by the client to resume receiving messages from the connection that was created in another TCP/client session. Agent response to this command can be `OK` or `ERR` in case connection does not exist (or can only be used to send connections - e.g. when the reply queue was not created).
#### SEND command and MID, SENT, RCVD and MERR responses
`SEND` command is used by the client to send messages.
`MID` response with the message ID (the sequential message number that includes both sent and received messages in the connection) is sent to the client to confirm that the message is accepted by the agent, before it is sent to the SMP server.
`SENT` notification is sent by the agent to confirm that the message was delivered to at least one of SMP servers. This notification contains the same message ID as `MID` notification. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session.
`RCVD` notification is sent by the agent when it receives `ACK` message from the receiving agent. This notification contains reception status, only one successful notification will be sent, and multiple error notifications will be sent in case `ACK` had error status.
In case of the failure to send the message for any other reason than network connection or message queue quota - e.g. authentication error (`ERR AUTH`) or syntax error (`ERR CMD error`), the agent will send to the client `MERR` notification with the message ID, and this message delivery will no longer be attempted to this SMP queue.
#### MSG notification
It is sent by the agent to the client when agent receives the message from the SMP server. It has message ID and timestamp from both the receiving and sending agents and from SMP server:
- recipient agent ID is intended to be used to refer to the message in the future.
- sender agent ID is intended to be used to identify any missed / skipped message(s)
- broker ID should be used to detect duplicate deliveries (it would happen if TCP connection is lost before the message is acknowledged by the agent - see [SMP protocol](./simplex-messaging.md))
#### END notification
It is sent by the agent to the client when agent receives SMP protocol `END` notification from SMP server. It indicates that another agent has subscribed to the same SMP queue on the server and the server terminated the subscription of the current agent.
#### DOWN and UP notifications
These notifications are sent when server or network connection is, respectively, `DOWN` or back `UP`.
All the subscriptions made in the current client session will be automatically resumed when `UP` notification is received.
#### OFF command
It is used to suspend the receiving SMP queue - sender will no longer be able to send the messages to the connection, but the recipient can retrieve the remaining messages. Agent response to this command can be `OK` or `ERR`. This command is irreversible.
#### DEL command
It is used to delete the connection and all messages in it, as well as the receiving SMP queue and all messages in it that were remaining on the server. Agent response to this command can be `OK` or `ERR`. This command is irreversible.
## Connection request
Connection request `connectionRequest` is generated by SMP agent in response to `newCmd` command (`"NEW"`), used by another party user with `joinCmd` command (`"JOIN"`), and then another connection request is sent by the agent in `replyQueueMsg` and used by the first party agent to connect to the reply queue (the second part of the process is invisible to the users).
@@ -407,12 +364,112 @@ smpQueue = <URL-encoded queueURI defined in SMP protocol>
All parameters are passed via URI hash to avoid sending them to the server (in case "https" scheme is used) - they can be used by the client-side code and processed by the client application. Parameters `smp` and `e2e` can be present in any order, any unknown additional parameters SHOULD be ignored.
`clientAppServer` is not an SMP server - it is a server that shows the instruction on how to download the client app that will connect using this connection request. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device.
`clientAppServer` is not an SMP server - it is a server that shows the instruction on how to download the client app that will connect using this connection link. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device.
"simplex" URI scheme in `connectionProtocol` can be used instead of client app server, to connect without creating any web traffic. Client apps MUST support this URI scheme.
See SMP protocol [out-of-band messages](./simplex-messaging.md#out-of-band-messages) for syntax of `queueURI`.
## Appendix A: SMP agent API
The exact specification of agent library API and of the events that the agent sends to the client application is out of scope of the protocol specification.
The list of some of the API functions and events below is supported by the reference implementation, and they are likely to be required by the client applications.
### API functions
The list of APIs below is not exhaustive and provided for information only. Please consult the source code for more information.
#### Create conection
`createConnection` api is used to create a connection - it returns the connection link that should be sent out-of-band to another protocol user (the joining party). It should be used by the client of the agent that initiates creating a duplex connection (the initiating party).
This api is also used to create a contact address - a special connection that can be used by multiple people to connect to the user.
Some communication scenarios may require fault-tolerant mechanism of creating connections that retries on network failures and continue retrying after the client is restarted. Such asynchronous API would return its result via `INV` event once it succeeds.
#### Join connection
`joinConnection` is used to create a connection record and accept the connection invitation received out-of-band. It should be used by the client of the agent that accepts the connection (the joining party).
This api can also be required as asynchronous, in which case `OK` event will be dispatched to the client to indicate the success or `ERR` in case it permanently failed (e.g., in case connection was deleted by another party).
#### Allow connection
Once the client receives `CONF` event, it should use synchronous `allowConnection` api to proceed with the connection (both for the [standard](#duplex-connection-procedure) and for the [fast duplex procedure](#fast-duplex-connection-procedure)).
In case this API is used as asynchronous it will return its result via `OK` or `ERR` event.
#### Accept and reject connection requests
Connection requests are delivered to the client application via `REQ` event.
Client can `acceptContact` and `rejectContact`, with `OK` and `ERR` events in case of asynchronous calls.
#### Send message
`sendMessage` api is always asynchronous. The api call returns message ID, `SENT` event once the message is sent to the server, `MWARN` event in case of temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client) and `MERR` in case of permanent delivery failure.
#### Acknowledge received message
Messages are delivered to the client application via `MSG` event.
Client application must always `ackMessage` to receive the next one - failure to call it in reference implementation will prevent the delivery of subsequent messages until the client reconnects to the server.
This api is also used to acknowledge message delivery to the sending party - that party client application will receive `RCVD` event.
#### Subscribe connection
`subscribeConnection` api is used by the client to resume receiving messages from the connection that was created in another TCP/client session.
#### Get notification message
`getNotificationMessage` is used by push notification subsystem of the client application to receive the message from a specific messaging queue mentioned in the notification. The client application would receive `MSG` and any other events from the agent, and then `MSGNTF` event once the message related to this notification is received.
#### Rotate message queue to another server
`switchConnection` api is used to rotate connection queues to another messaging server.
#### Renegotiate e2e encryption
`synchronizeRatchet` api is used to re-negotiate double ratchet encryption for the connection.
#### Delete connection
`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQ` and `DEL_CONN` events.
#### Suspend connection
`suspendConnection` api is used to prevent any further messages delivered to the connection without deleting it.
### API events
Agent API uses these events dispatch to notify client application about events related to the connections:
- `INV` - connection invitation or connection address URI after connection is created.
- `CONF` - confirmation that connection is accepted by another party. When the accepting party uses `joinConnection` api to accept connection invitation, the initiating party will receive `CONF` notification with some identifier and additional information from the accepting party (e.g., profile). To continue the connection the initiating party client should use `allowConnection` api.
- `REQ` - connection request is sent when another party uses `joinConnection` api with contact address. The client application can use `acceptContact` or `rejectContact` api.
- `INFO` - information from the party that initiated the connection with `createConnection` sent to the party accepting the connection with `joinConnection`.
- `CON` - notification that connection is established sent to both parties of the connection.
- `END` - notification that connection subscription is terminated when another client subscribed to the same messaging queue.
- `DOWN` - notification that connection server is temporarily unavailable.
- `UP` - notification that the subscriptions made in the current client session are resumed after the server became available.
- `SWITCH` - notification about queue rotation process.
- `RSYNC` - notification about e2e encryption re-negotiation process.
- `SENT` - notification to confirm that the message was delivered to at least one of SMP servers. This notification contains the same message ID as returned to `sendMessage` api. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session.
- `MWARN` - temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client).
- `MERR` - notification about permanent message delivery failure.
- `MERRS` - notification about permanent message delivery failure for multiple messages (e.g., when multiple messages expire).
- `MSG` - sent when agent receives the message from the SMP server.
- `MSGNTF` - sent after agent received and processed the message referenced in the push notification.
- `RCVD` - notification confirming message receipt by another party.
- `QCONT` - notification that the agent continued sending messages after queue capacity was exceeded and recipient received all messages.
- `DEL_RCVQ` - confirmation that message queue was deleted.
- `DEL_CONN` - confirmation that connection was deleted.
- `OK` - confirmation that asynchronous api call was successful.
- `ERR` - error of asynchronous api call or some other error event.
This list of events is not exhaustive and provided for information only. Please consult the source code for more information.
note over B: having received file description<br>from sender
loop for each chunk
B ->> S: 1a. download chunk ("FGET")
S ->> B: send chunk body ("FILE")
opt
B ->> S: 1b. acknowledge chunk reception ("FACK")
note over S: delete recipient ID
S ->> B: respond with ok ("OK")
end
end
note over B: 2. combine chunks into a file<br>3. decrypt file using key from file description<br>4. extract file name and unpad the file<br>5. validate file digest with the file description
note over CI, HI: 3. Session verification and protocol negotiation
HC->>CC: host HELLO
note over CC: validate version, CA fingerprint
alt
CC-->>HC: controller ERROR
else
CC-->>HC: controller HELLO
note over CC, HC: update stored keys
end
deactivate CC
note over CI, HI: 4. Session operation
loop
CI->>CC: command
CC->>HC: XRCP command
HC-->>CC: XRCP response
CC-->>CI: response
end
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.