783 Commits

Author SHA1 Message Date
Evgeny
a1b762992b agent: pass key and link ID when preparing group link (#1754)
* agent: pass key and link ID when preparing group link

* binding

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-03-28 20:29:48 +00:00
sh
efcef2d1fd xftp-web: add postgres schema cleanup for integration tests (#1741)
Stale postgres schema leaked pending XFTP operations between
cross-language tests, causing N-1 of N tests to fail.
2026-03-20 09:58:56 +00:00
sh
dc2921e4ce xftp-server: embed file download widget in XFTP server web page (#1733)
* xftp-server: embed file download widget in XFTP server web page

When a URL has a hash fragment (>50 chars), the server page shows the
file download UI instead of the server info page. Embeds xftp-web
assets (JS, CSS, crypto worker) and protocol overlay with matching
website content. Overlay renders below the server navbar.

* xftp-server: fix overlay scroll lock, remove extra margin, fix dark SVG

* xftp-server: move file transfer widget to standalone /file page

* web: collapse all repeated Nothing sections in render

section_ only collapsed the first occurrence of a section when content
was Nothing, leaving subsequent sections with the same label intact.
This caused SMP server pages to show raw <x-xftpConfig> tags.

* xftp-server: update bundled css/js

* xftp-server: move file.html to xftp-server, rename xftp bundle dir

* web: remove unused server-info wrapper div

* refactor

* fix

---------

Co-authored-by: Evgeny <evgeny@poberezkin.com>
2026-03-13 16:00:02 +00:00
sh
3a627a9599 tests: use correct web CA certificate in CLI static files test (#1727) 2026-03-09 12:23:03 +00:00
sh
437cdde4a5 xftp: add web page for server information (#1724)
* xftp: add web page for server information

* web: rename XFTP.Web to XFTPWeb, remove XFTP subdirectory

* refactor(xftp): remove storage quota from web page

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-09 08:44:28 +00:00
sh
eed1bf14c6 web: extract shared web module from smp-server (#1723)
* web: extract shared web module from smp-server

Move web serving infrastructure (warp, static files, HTML templating)
from apps/smp-server/web/Static.hs into library modules:
- Simplex.Messaging.Server.Web (generic web infra + templating)
- Simplex.Messaging.Server.Web.Embedded (TH-embedded assets)

Move static assets from apps/smp-server/static/ to
src/Simplex/Messaging/Server/Web/.

Move EmbeddedWebParams/WebHttpsParams from Server.Main to Server.Web.

Keep SMP-specific rendering (serverInformation) in apps/smp-server/SMP/Web.hs.

generateSite is now generic: takes pre-rendered HTML + link page paths,
enabling reuse by XFTP and NTF servers.

* web: add tests for templating engine

Tests for render, section_, item_, and timedTTLText functions
in Simplex.Messaging.Server.Web module.

* web: add serverInfoSubsts, serveStaticPageH2, safe port parsing

* web: rename SMP.Web to SMPWeb, remove SMP subdirectory

* fix(web): section_ collapsing sections with Just "" content

Commit e48bedea ("servers: fix server pages when source code is not
specified") changed section_ to treat Just "" the same as Nothing -
collapsing the section. The intent was to handle the sourceCode case
(empty string when not specified), but the guard
`not (B.null content)` also broke operator, admin, complaints, and
hosting - all of which legitimately use Just "" as a
section-present marker.

Before (correct):
  Nothing -> before <> next
  Just content -> before <> item_ label content inside <> ...

After (broken):
  Just content | not (B.null content) -> ...
  _ -> before <> next

Restore the original behavior: only Nothing collapses a section.

* refactor

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-03-09 08:42:38 +00:00
Evgeny
f6aca47604 xftp: implementation of XFTP client as web page (#1708)
* xftp: implementation of XFTP client as web page (rfc, low level functions)

* protocol, file descriptions, more cryptogrpahy, handshake encoding, etc.

* xftp server changes to support web slients: SNI-based certificate choice, CORS headers, OPTIONS request

* web handshake

* test for xftp web handshake

* xftp-web client functions, fix transmission encoding

* support description "redirect" in agent.ts and cross-platform compatibility tests (Haskell <> TypeScript)

* rfc: web transport

* client transport abstraction

* browser environment

* persistent client sessions

* move rfcs

* web page plan

* improve plan

* webpage implementation (not tested)

* fix test

* fix test 2

* fix test 3

* fixes and page test plan

* allow sending xftp client hello after handshake - for web clients that dont know if established connection exists

* page tests pass

* concurrent and padded hellos in the server

* update TS client to pad hellos

* fix tests

* preview:local

* local preview over https

* fixed https in the test page

* web test cert fixtures

* debug logging in web page and server

* remove debug logging in server/browser, run preview xftp server via cabal run to ensure the latest code is used

* debug logging for page sessions

* add plan

* improve error handling, handle browser reconnections/re-handshake

* fix

* debugging

* opfs fallback

* delete test screenshot

* xftp CLI to support link

* fix encoding for XFTPServerHandshake

* support redirect file descriptions in xftp CLI receive

* refactor CLI redirect

* xftp-web: fixes and multi-server upload (#1714)

* fix: await sodium.ready in crypto/keys.ts (+ digest.ts StateAddress cast)

* multi-server parallel upload, remove pickRandomServer

* fix worker message race: wait for ready signal before posting messages

* suppress vite build warnings: emptyOutDir, externals, chunkSizeWarningLimit

* fix Haskell web tests: use agent+server API, wrap server in array, suppress debug logs

* remove dead APIs: un-export connectXFTP, delete closeXFTP

* fix TypeScript errors in check:web (#1716)

- client.ts: cast globalThis.process to any for browser tsconfig,
  suppress node:http2 import, use any for Buffer/chunks, cast fetch body
- crypto.worker.ts: cast sha512_init() return to StateAddress

* fix: serialize worker message processing to prevent OPFS handle race

async onmessage allows interleaved execution at await points.
When downloadFileRaw fetches chunks from multiple servers in parallel,
concurrent handleDecryptAndStore calls both see downloadWriteHandle
as null and race on createSyncAccessHandle for the same file,
causing intermittent NoModificationAllowedError.

Chain message handlers on a promise queue so each runs to completion
before the next starts.

* xftp-web: prepare for npm publishing (#1715)

* prepare package.json for npm publishing

Remove private flag, add description/license/repository/publishConfig,
rename postinstall to pretest, add prepublishOnly, set files and main.

* stable output filenames in production build

* fix repository url format, expand files array

* embeddable component: scoped CSS, dark mode, i18n, events, share

- worker output to assets/ for single-directory deployment
- scoped all CSS under #app, removed global resets
- dark mode via .dark ancestor class
- progress ring reads colors from CSS custom properties
- i18n via window.__XFTP_I18N__ with t() helper
- configurable mount element via data-xftp-app attribute
- optional hashchange listener (data-no-hashchange)
- completion events: xftp:upload-complete, xftp:download-complete
- enhanced file-too-large error mentioning SimpleX app
- native share button via navigator.share

* deferred init and runtime server configuration

- data-defer-init attribute skips auto-initialization
- window.__XFTP_SERVERS__ overrides baked-in server list

* use relative base path for relocatable build output

* xftp-web: retry resets to default state, use innerHTML for errors

* xftp-web: only enter download mode for valid XFTP URIs in hash

* xftp-web: render UI before WASM is ready

Move sodium.ready await after UI initialization so the upload/download
interface appears instantly. WASM is only needed when user triggers
an actual upload or download. Dispatch xftp:ready event once WASM loads.

* xftp-web: CLS placeholder HTML and embedder CSS selectors

Add placeholder HTML to index.html so the page renders a styled card
before JS executes, preventing layout shift. Use a <template> element
with an inline script to swap to the download placeholder when the URL
hash indicates a file download. Auto-compute CSP SHA-256 hashes for
inline scripts in the vite build plugin.

Change all CSS selectors from #app to :is(#app, [data-xftp-app]) so
styles apply when the widget is embedded with data-xftp-app attribute.

* xftp-web: progress ring overhaul

Rewrite progress ring with smooth lerp animation, green checkmark on
completion, theme reactivity via MutationObserver, and per-phase color
variables (encrypt/upload/download/decrypt).

Show honest per-phase progress: each phase animates 0-100% independently
with a ring color change between phases. Add decrypt progress callback
from the web worker so the decryption phase tracks real chunk processing
instead of showing an indeterminate spinner.

Snap immediately on phase reset (0) and completion (1) to avoid
lingering partial progress. Clean up animation and observers via
destroy() in finally blocks.

* xftp-web: single progress ring for upload, simplify ring color

* xftp-web: single progress ring for download

* feat(xftp-web): granular progress for encrypt/decrypt phases

Add byte-level progress callbacks to encryptFile, decryptChunks,
and sha512Streaming by processing data in 256KB segments. Worker
reports fine-grained progress across all phases (encrypt+hash+write
for upload, read+hash+decrypt for download). Progress ring gains
fillTo method for smooth ease-out animation during minimum display
delays. Encrypt/decrypt phases fill their weighted regions (0-15%
and 85-99%) with real callbacks, with fillTo covering remaining
time when work finishes under the 1s minimum for files >= 100KB.

* rename package

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

---------

Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
Co-authored-by: shum <github.shum@liber.li>
Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2026-03-02 09:57:46 +00:00
Evgeny Poberezkin
d7b90b8415 Merge branch 'stable' 2026-01-28 22:52:33 +00:00
Evgeny
3c5ec8d9a1 agent: improve error handling (#1707)
* agent: improve error handling

* simplify

* report critical error when subscriber crashes

* fix test
2026-01-28 21:54:41 +00:00
Evgeny
d10e05b796 agent: split creating connection to two steps to prepare connection link in advance (#1704)
* agent: split creating connection to two steps to prepare connection link in advance

* linkEntityId, newOwnerAuth

* simplify
2026-01-27 10:54:13 +00:00
Evgeny
66cc06738e agent: fix tests for short links (#1703) 2026-01-26 00:11:50 +00:00
spaced4ndy
89b81d151f agent: getConnShortLinkAsync; implement joinConnectionAsync for contact connections; narrow setConnShortLinkAsync only to Contact connections (#1694)
* agent: getConnShortLinkAsync

* enable all tests

* comment

* comment

* join conn async for contact URI (wip)

* fix test

* remove enableNtfs param

* FixedLinkData

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-01-20 08:40:35 +00:00
Evgeny
3e5b654109 agent: support multiple link owners in link data (#1701)
* agent: support multiple link owners in link data

* fix
2026-01-19 22:08:11 +00:00
Evgeny
1000107259 agent: clean up old records in batches (#1692)
* agent: clean up old records in batches

* update

* update

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* stabilize tests

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2026-01-13 19:11:46 +00:00
Evgeny
6aadcf1f3f agent: lock rows for concurrent queries in PostgreSQL (#1688)
* agent: lock rows for concurrent queries in PostgreSQL

* fix race conditions in workers

* refactor
2026-01-08 11:09:58 +00:00
spaced4ndy
07604a146f agent: async command to set connection short link (setConnShortLinkAsync) (#1680) 2026-01-07 15:12:52 +00:00
Evgeny
a7b43b1a3e agent: use strict tables (#1686)
* agent: use strict tables

* migrate existing tables to strict

* test: verify that all tables are strict

* fix column types for device_token and ntf_mode

* fix encodings and column types for ntf_sub_action and ntf_sub_smp_action

* update schema

* remove debug.trace

* log
2026-01-03 17:19:18 +00:00
Evgeny
d6df769799 ntf server: additional tests (#1687) 2026-01-02 14:07:13 +00:00
Evgeny Poberezkin
70d1b99fb4 Merge branch 'stable' 2025-12-23 14:18:07 +00:00
Evgeny
bbe1c716e6 xrcp: treat any 127.x.x.x IP address as local (#1682) 2025-12-23 12:42:05 +00:00
Evgeny
cf9b7e5b6a agent: option to add SQLite functions to DB connection (#1674)
* agent: option to add SQLite functions to DB connection

* add module
2025-12-03 16:10:35 +00:00
spaced4ndy
3016b929b4 agent: make createConnection and setConnShortLink apis support setting all link data fields (#1665)
* agent: make createConnection and setConnShortLink apis support setting all link data fields

* add functions

* refactor

* refactor

* fix tests

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-11-05 07:08:33 +00:00
Evgeny
1329fc726f smp: support client notices (#1659)
* agent: support client notices

* improve

* fix, test

* rename

* cleanup

* send and process notices in more cases

* dont delete

* dont remove notice on other permanent errors

* dont remove notice if there is no notice ID in queue

* add server to error

* allow deleting

* only use notice if key hash matches
2025-10-17 18:34:59 +01:00
Evgeny
80aa56cbcc agent: track which queues need subscribing for iOS NSE (#1657)
* agent: track which queues need subscribing for iOS NSE

* fix down migration

* fix, cleanup
2025-10-09 18:33:27 +01:00
Evgeny
9cda20381f agent: subscribe all connections (#1655)
* agent: subscribe all connections

* query, version

* BoolInt

* add query to errors

* Revert "add query to errors"

This reverts commit 32a1f7fe11.

* fix optional field

* version

* limit number of in-flight subscriptions to 35000
2025-10-09 13:43:48 +01:00
Evgeny
11a4859f8e agent: batch processing of subscription results and errors (#1652)
* agent: batch processing of subscription results and errors

* run agent tests with in-memory server storage

* version

* non empty errors

* size
2025-10-07 14:49:38 +01:00
Evgeny
a3d1a72eb0 agent: optimize subscriptions memory usage (#1651)
* agent: optimize subscriptions memory usage more (do not store subscribed queues in memory) WIP

* use new session subscriptions data

* version

* remove old data structure

* remove version

* batch deletions

* test TSessionSubs

* comment
2025-10-06 15:21:17 +01:00
Evgeny
779222d1a7 agent: optimize subscriptions memory usage (#1649)
* agent: optimize subscriptions memory usage

* version

* remove typeclass

* disable subscription to test memory usage

* disable delivery and commands

* disable reading connections

* enable subscriptions

* RcvQueueCred

* RcvQueueCred postgres

* fix

* optimize keys

* version

* rename

* comments
2025-10-05 13:35:54 +01:00
Evgeny
99f40ae109 support for additional database migrations (#1644) 2025-09-27 11:21:38 +01:00
Evgeny
7b7616ce7e agent: batch loading of connections with PostgreSQL client (#1639)
* agent: batch loading of connections with PostgreSQL client

* batch more

* optimize getPendingCommandServers

* fix Bool conversion

* enable all tests

* cleanup
2025-09-16 14:28:06 +01:00
Evgeny
d3754b36bc smp server: do not use queue cache with PostgreSQL message storage (#1637)
* smp server: do not use queue cache with PostgreSQL message storage

* fix loading queues via notifier IDs
2025-09-15 07:51:27 +01:00
Evgeny Poberezkin
112cd9d5f4 Merge branch 'stable' 2025-09-14 14:42:05 +01:00
Evgeny
8a7991a376 agent: option to set SQLite database backup path (#1636)
* agent: option to set SQLite database backup path

* fix test compilation
2025-09-14 12:41:49 +01:00
Evgeny
9be9a76b42 smp server: improve message expiration (#1634)
* smp server: limit by time the queues to export journal messages for

* pass queue/msg thresholds separately

* reset db connection on errors

* Revert "smp server: limit by time the queues to export journal messages for"

This reverts commit d3bc0cba4b.

* fix test compilation

* flag to expire messages

* improve test

* expire messages newer than quota
2025-09-13 11:35:34 +01:00
Evgeny
a137d01c90 smp server: improve counts management and expiration for postgres message store (#1631)
* smp server: improve counts management and expiration for postgres message store

* one update

* fix expiration
2025-09-12 11:39:35 +01:00
Evgeny
bac6ea6e91 smp server: store messages in PostgreSQL (#1622)
* smp server: store messages in PostgreSQL

* stored procedures to write and to expire messages

* function to export messages

* move all message functions to PostgreSQL, remove delete trigger

* comments

* import messages to db

* fix message import, add export

* fix export

* fix export

* fix compilation flags

* import messages line by line

* fix server start with database storage

* fix compilation

* comments
2025-09-11 20:22:55 +01:00
Evgeny
0c1030cf02 smp server: faster export (#1626)
* smp server: faster export

* flush

* export messages with PostgreSQL database

* remove flush

* batch message writes

* Revert "batch message writes"

This reverts commit 61fb3c3b9e.

* remove $!

* fast journal export

* another approach

* Revert "smp server: remove dependency of message size on the version (#1627)"

This reverts commit 8fea15245a.

* style

* faster?

* cleanup

* cleanup

* refactor

* refactor

* concurrent read messages

* Revert "concurrent read messages"

This reverts commit 05a32e6491.

* concurrent read/write

* parameter to export to/import from another message log file

* Revert "parameter to export to/import from another message log file"

This reverts commit 4e88b035c4.

* Revert "concurrent read/write"

This reverts commit a8eab1f7e3.
2025-09-10 18:22:58 +01:00
Evgeny Poberezkin
23aff6b0b1 Revert "smp server: remove dependency of message size on the version (#1627)"
This reverts commit 8fea15245a.
2025-09-09 09:56:48 +01:00
Evgeny
8fea15245a smp server: remove dependency of message size on the version (#1627) 2025-09-08 11:28:42 +01:00
Evgeny
4386dd8b5a smp server: option to expire messages when new message is sent (now off by default) (#1625)
* smp server: option to expire messages when new message is sent (now off by default)

* fix test
2025-09-07 09:02:32 +01:00
Evgeny
cb3250e7b4 servers: better socket leak prevention during TLS handshake, add NetworkError type to better diagnose connection errors (#1619)
* servers: better socket leak prevention during TLS handshake

* log tcp connection errors

* more detailed network error

* log full address

* rename error

* add encodings for NetworkError

* refactor

* comment

* bind

* style

* remove parameters of NETWORK error from encoding
2025-09-02 16:07:37 +01:00
Evgeny
77bf76e1f9 smp server: fix in-memory server not restoring queue/service associations after 2+ restarts (#1618)
* smp server: fix in-memory server not restoring queue/service associations after 2+ restarts

* fix test, do not reuse database
2025-09-01 17:45:31 +01:00
Evgeny
beafac1f73 agent: make agent workers usable from other contexts (#1614) 2025-08-29 08:33:55 +01:00
Evgeny
e48bedeaf2 servers: fix server pages when source code is not specified, include commit SHA in printed version and in web page. (#1608)
* smp server: fix server pages when source code is not specified

* servers: include git commit in version

* flexible alpn

* fix test

* fix ghc 8.10.7 build
2025-08-23 19:59:00 +01:00
Evgeny Poberezkin
4b7fc34fe3 enable all tests 2025-08-17 10:40:12 +01:00
Evgeny
96e8b4a146 smp server: wrap all queries in transactions (#1603)
* smp server: wrap all queries in transactions

* fix test

* fix schema test
2025-08-17 10:38:20 +01:00
Evgeny
f4e7469f96 xftp server: prometheus metrics (#1595)
* xftp server: prometheus metrics

* update

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>

---------

Co-authored-by: sh <37271604+shumvgolove@users.noreply.github.com>
2025-07-30 16:18:49 +01:00
Evgeny
c0f357d817 servers: control port session improvements (#1591)
* servers: prohibit changing role during control port session

* quota for blocked queues

* allow disabling blocking and quota

* fix test

* fix INI file
2025-07-21 15:19:43 +01:00
Evgeny
40fc09a93d smp server: additional stat counter for ntf credentials created together with the queue (#1589)
* smp server: additional stat counter for ntf credentials created together with the queue

* fix prometheus

* fix test

* fix qSub
2025-07-18 09:26:25 +01:00
Evgeny
2a90a2c552 smp protocol: create notification credentials via NEW command that creates the queue (#1586)
* smp protocol: create notification credentials via NEW command that creates the queue

* create ntf subscription for queues created with ntf credetials

* do not create ntf credentials when switching connection to another queue
2025-07-16 16:46:45 +01:00