docs: add smp-server memory findings

This commit is contained in:
shum
2026-09-30 23:56:47 +00:00
parent 84d91bd445
commit 4b217d618f
2 changed files with 427 additions and 0 deletions
+243
View File
@@ -0,0 +1,243 @@
# SMP server memory at production scale
Production: ~40k client connections per server, 32 GB RAM shared with PostgreSQL, pure PostgreSQL
store, 14 servers proxying to each other, GHC 9.6.3, `+RTS -N -F1.2 -A16m -I0.01 -Iw15 -s -RTS`.
With `-F1.2` the copying GC needs about 2.2x the live heap plus the nursery, so the server's live
heap has to stay well under ~10 GB.
A busy connection (20 batch-subscribed queues, 2 created with NEW, message traffic) cost 237 KiB of
server live heap, 9.0 GiB at 40k connections. With the changes below it costs 43 KiB, 1.6 GiB.
Earlier findings on leaks and PostgreSQL load are in `leak-findings.md` and `smp-server-db-load.md`.
## Method
All numbers are server-only: the bench client runs in a child process, so the measuring process holds
only the server. PostgreSQL store, production RTS flags, 16-core machine. Live heap is measured after
two major GCs with finalizers allowed to run in between (`liveBytesMiB` in `bench/MemBench.hs`).
| Phase | Measures |
| --- | --- |
| `prodmix` | 2000 connections, each SUBs 20 queues in one block and creates 2 with NEW, then 30 s of SEND/MSG/ACK traffic |
| `conns N` | N idle connections after the SMP handshake |
| `subslice N` | per subscription: `SUBMODE=sub` (`SUBBATCH`, `SUBKEEP`), `new`, `newthensub` |
| `load` | server CPU per operation under a fixed mixed workload |
| `ntfloop`, `ntfdeliver` | notification store and its delivery loop |
| `mesh` | proxy and relay in separate processes: relay sessions, forwards in flight, forward rate (`MESH=`) |
| `pfwdbig` | oversized forwards left in the proxy's `sentCommands` |
```sh
cabal build -fserver_postgres exe:smp-mem-bench
BENCHID=1 PROD_CONNS=2000 PROD_QUEUES=20 PROD_NEW=2 PROD_SEC=30 \
$(cabal list-bin -fserver_postgres exe:smp-mem-bench) prodmix 0 \
+RTS -N -F1.2 -A16m -I0.01 -Iw15 -T -ki2k -RTS
```
`BENCHID` offsets ports and the database, so runs do not collide with each other or with tests.
## Results
`prodmix`, live heap per connection after subscribing and peak memory in use under traffic. Each row
adds to the previous one.
| Change | KiB/conn | Peak in use | 40k conns |
| --- | --- | --- | --- |
| base (`sh/fix-leak`) | 236.6 | 1075 MiB | 9.0 GiB |
| `-ki2k` | 111.5 | 719 MiB | 4.3 GiB |
| connection commits (`sh/fix-conn-mem`) | 89.8 | 635 MiB | 3.4 GiB |
| unpinned subscription keys, proxy commits (`sh/mem-combined`) | 80.0 | 604-622 MiB | 3.1 GiB |
| patched tls 1.9 | 42.9 | 469 MiB | 1.6 GiB |
Idle connections (`conns 2000`): 153 KiB at default flags, 68.4 KiB on `sh/mem-combined` with
`-ki2k`, 34.8 KiB with the patched tls.
Message throughput did not drop in any row. `-ki2k` costs some GC time (below).
## Findings
### 1. Thread stacks: 125 KiB per busy connection
Each connection has several threads that block in shallow loops. A thread starts with a 1 KiB stack
(`-ki1k`). On first overflow the RTS copies up to `-kb` (1 KiB) of frames into a new 32 KiB chunk
(`-kc32k`), which with a 1 KiB initial stack moves the loop frame itself, so the thread never returns
to its first chunk and keeps the 32 KiB chunk while the connection lives.
| Flags | idle KiB/conn | prodmix KiB/conn |
| --- | --- | --- |
| default | 153 | 236.6 |
| `-kc8k` | 100 | 135.0 |
| `-ki2k` | 91 | 111.5 |
`-ki2k` leaves the loop frame in the first chunk, so the overflow chunk is freed on return. Cost, 3
alternating 60 s `load` runs: GC CPU +15% (4.8-5.2 s to 5.7-5.8 s per run), major GCs +43% (the live
heap is smaller, so `-F1.2` triggers major GCs sooner), CPU per operation +0-5% (2.74-3.02 ms to
2.91-3.35 ms, one noisy run). One `load` run with `-kc2k` grew to 6.8 GB RSS; not investigated, so
`-kc` is left at its default.
Branch `sh/rts-ki2k` sets `-with-rtsopts=-ki2k` for smp-server. Command-line `+RTS` flags still apply.
### 2. Pinned blocks kept alive by small long-lived objects
`ByteString`s and crypton's `ScrubbedBytes` are pinned and share 4 KiB blocks. GHC treats a pinned
block as live if any object in it is live, and treats every object in a live pinned block as alive,
including dead `ScrubbedBytes`, whose weak pointers and finalizers then stay allocated (GHC 9.6.3
`Evac.c:443`, `GCAux.c:73`). A 24-byte value created during request processing, among crypto
temporaries, therefore keeps ~4 KiB and several dead keys alive for as long as it lives. A standalone
program reproduces it (pinned key 2742 B live per key, unpinned 80 B).
Subscriptions. The subscription key is such a value.
| `subslice 5000` | before | unpinned keys |
| --- | --- | --- |
| NEW-created | 5.25 KiB | 0.46 KiB |
| one SUB per block | 1.67-2.42 KiB | 0.47 KiB |
| 100 SUBs per block | 0.49 KiB | 0.46 KiB |
| NEW0, then SUB on the same connection | 4.98 KiB | 0.47 KiB |
Branch `sh/fix-sub-mem` stores the keys of `subscriptions`, `ntfSubscriptions` and `queueSubscribers`
as `ShortByteString` and shares one key between the maps. CPU unchanged.
TLS session state. tls 1.9 keeps record keys, IVs, TLS 1.3 traffic secrets, verify data and
handshake state for the connection's lifetime, all allocated during the handshake. A heap census per
connection before and after re-allocating them together at the end of the handshake:
| | tls 1.9 | patched |
| --- | --- | --- |
| weak pointers | ~138 | ~45 |
| memory in pinned blocks, outside the census | ~39 KiB | ~13 KiB |
| idle KiB/conn | 68.4 | 34.8 |
| prodmix KiB/conn | 80.0 | 42.9 |
The patch (`tls-1.9-compact-state.patch`: `Network.TLS.Handshake.Compact`, called at the end of
`handshake` under the context locks) re-derives the TLS 1.3 record keys from the kept traffic
secrets, copies the IVs, secrets, session ID, verify data, randoms, main and resumption secrets and
the transcript hash, and reseeds the context RNG. It needs a tls fork. Not compacted: the TLS 1.2 bulk
key, keys installed later by KeyUpdate. The transcript hash copy coerces crypton's hidden `Context`
newtype, so it must be checked on crypton upgrades.
Full test suite with the patched tls: 1431 examples, 0 failures. The same run on unpatched tls had 1
failure, the XFTP agent's "should resume sending file after restart".
SMP handshake state (session secret, chain keys) has the same pattern and is not done yet; the
remaining ~13 KiB outside the census is likely there.
### 3. IDs sliced from the received block: 16 KiB per subscription
Parsed `corrId` and `entityId` are slices of the ~16 KB decrypted block, and the entity ID was stored
as the subscription key, so one live key kept the block. One SUB per block cost 16.43 KiB per
subscription, one survivor of 100 cost 18.04 KiB. `sh/fix-sub-keys` copies both IDs in
`tDecodeServer` (16.43 to 1.67-2.42 KiB). Unpinned keys (finding 2) also remove it for the maps.
### 4. Connection structure: 21 KiB per busy connection
- tls 1.9 stores the receive record state lazily, so each connection kept its last received 16 KB
record. `recvTLS` forces the state after each read. Not needed with tls 2.x.
- The send and message-send loops are one thread, and one server-wide thread expires inactive clients
instead of one per client: 6 threads per connection become 4.
`sh/fix-conn-mem`: `prodmix` 111.5 to 89.8 KiB with `-ki2k`, 236.6 to 210.5 KiB without.
### 5. Notification store and its delivery loop
Keys of notifiers whose notifications were delivered stayed in the store, and the loop in
`deliverNtfsThread` sends every key to `getQueueNtfServices` (one `notifier_id IN ?` query read in
full) every 1.5 s.
| `ntfloop` (keys, no traffic) | allocation | CPU | peak in use |
| --- | --- | --- | --- |
| 0 | 0 MiB/s | 0 | 280 MiB |
| 100k | 168 MiB/s | 0.42 cores | 530 MiB |
| 300k | 263 MiB/s | 0.76 cores | 913 MiB |
`ntfdeliver` with 20k delivered notifications: 20000 keys kept, idle server allocating 36 MiB/s at
0.18 cores; with `sh/fix-ntf-store` 0 keys, 0.2 MiB/s, 0.01 cores.
### 6. Proxy mesh
The relay processed forwarded commands from a proxy one at a time in the connection's command loop,
with a PostgreSQL round trip each; all users of a proxy share its connection. Forwards the relay has
not reached wait on the proxy, each with a thread and its ~16 KB block. `mesh`, 1000 forwards per
second for 20 s, proxy and relay in separate processes, 2 runs each:
| | serial | concurrent |
| --- | --- | --- |
| forwards ok / failed | 19,968 / 0 | 19,968 / 0 |
| proxy live heap at end of sending | 192-200 MiB | 13 MiB |
| proxy memory in use | 653-672 MiB | 325 MiB (idle baseline) |
| proxy threads | 4,026-4,137 | 420 |
Here the backlog drained within the 30 s forward timeout. With a slower relay (a busy database, a
remote relay over SOCKS) the queueing delay exceeds it and forwards fail; not reproduced.
`sh/fix-proxy-mesh` forks each forwarded command under the per-connection concurrency limit, stops
keeping the forwarded block on the proxy while waiting (53.0 to 36.8 KiB per in-flight forward,
relay not answering) and caps forwards in flight per relay at 512 (`[PROXY] relay_concurrency`).
Forwarded commands use per-command nonces and secrets and are matched by correlation ID, so concurrent
processing is safe; the SimpleX agent keeps one message in flight per queue. The spec requires
responses in order per queue within a connection (`simplex-messaging.md`, "same order within each
queue ID"); serializing forwarded commands per queue ID is not done yet.
Also in the proxy path (`sh/fix-proxy-leak`): a PFWD of 16260-16266 bytes from a client that declares
`proxyServer = True` failed with `TELargeMsg` before sending and left its 20.2 KiB request in
`sentCommands` for the life of the relay session (2001 PFWDs, 2001 entries); timed-out requests were
kept too; and a relay session that stopped answering was never dropped (10 of 10 forwards timed out).
### 7. Concurrency limit and name resolution
`forkCmd` released its slot when the thread was forked, not when the command finished, so
`serverClientConcurrency` and `serverResolverConcurrency` limited nothing: 16 RSLVs with a limit of 4
all reached the resolver, 64 RSLVs from one connection were 64 concurrent requests, and 5 lookups
answered with 502 opened 5 connections because error bodies were not read. `sh/fix-rslv-fanout` holds
the slot until completion, adds a global limit (`[NAMES] resolver_global_concurrency`, 32) and reads
error bodies. With the limit working, a client that hits it blocks its own command loop.
### 8. Smaller findings
- Active-queue statistics are `IntSet`s at 64 B per element (15M elements, 916 MiB for one set), six
of them, reset only when `log_stats` is on.
- Control port `save` closes the PostgreSQL pool while the server keeps running; every later DB
operation blocks (`cpsave` phase: NEW after `save` gets no response).
### 9. tls 2.x
tls 2.1.6 (with tls-session-manager 0.0.6, crypton-connection 0.4.3, http-client-tls 0.3.6.4, an
index-state bump and version pins) saves 2-5 KiB per connection: `prodmix` 75.4 KiB against 80.0.
Problems found:
- After a failed handshake a tls 2 client and a tls 2 server wait for each other in `bye`; this hung
`testServerMultipleIdentities`. Fixed on `sh/tls2` by closing the context without `bye`.
- A tls 2 client that receives a CertificateRequest runs a timed read inside `handshake`; when the
timeout fires in the middle of a record the stream fails ("bad record mac") or hangs. SMP servers
always request client certificates. 1-2 failures per 2000 concurrent connections; present in tls
2.4.3. A `conns 2000` run with tls 2 on both sides stopped at 53 connections.
tls 2.x is not recommended; `sh/tls2` is kept for reference.
## Open
- Serialize forwarded commands per queue ID on the relay (spec response order).
- A client that sends PFWDs and never reads the responses costs the proxy 6.3 MiB live (11.4 MiB in
use) until the inactive-client expiry (up to 6 hours), 6 GiB per 1000 such clients.
- PRXY to many aliases of one relay opens an unbounded number of relay sessions, each 207 KiB on the
proxy and 149 KiB on the relay: 2.0 GiB and 1.4 GiB per 10k.
- Compact the SMP handshake state after the handshake, as in finding 2.
- Publish the tls fork and reference it from `cabal.project`.
- A SEND right after NEW or SUB can find no subscriber yet (`queueSubscribers` is updated through
`subQ`), so the message waits for the next SUB or ACK. `deliverIfSame` may leave a subscription
without a delivery thread. Both seen as occasional stuck steps in `load`; not memory.
- `[PROXY] relay_concurrency` and `[NAMES] resolver_global_concurrency` are new INI keys.
## Branches
| Branch | Base | Content |
| --- | --- | --- |
| `sh/fix-ntf-store` | master | finding 5 |
| `sh/fix-sub-keys` | master | finding 3 |
| `sh/fix-proxy-leak` | master | finding 6, request leak and stuck session |
| `sh/fix-rslv-fanout` | master | finding 7 |
| `sh/fix-conn-mem` | master | finding 4 |
| `sh/fix-sub-mem` | master | finding 2, subscription keys |
| `sh/fix-proxy-mesh` | `sh/fix-proxy-leak` | finding 6, relay concurrency, per-relay limit |
| `sh/rts-ki2k` | master | finding 1 |
| `sh/mem-combined` | `sh/fix-leak` | findings 1-6 together, as measured in Results (local) |
| `sh/tls2` | `sh/mem-combined` | finding 9, not for merge (local) |
+184
View File
@@ -0,0 +1,184 @@
diff -ruN -x 'dist*' tls-1.9.0-orig/Network/TLS/Crypto.hs tls-fork/Network/TLS/Crypto.hs
--- tls-1.9.0-orig/Network/TLS/Crypto.hs 2026-09-30 18:44:40.083085887 +0000
+++ tls-fork/Network/TLS/Crypto.hs 2026-09-30 18:44:40.106519923 +0000
@@ -8,6 +8,7 @@
, hashUpdate
, hashUpdateSSL
, hashFinal
+ , hashCopy
, module Network.TLS.Crypto.DH
, module Network.TLS.Crypto.IES
@@ -44,6 +45,8 @@
import qualified Crypto.Hash as H
import qualified Data.ByteString as B
import qualified Data.ByteArray as B (convert)
+import qualified Data.ByteArray as BA
+import Unsafe.Coerce (unsafeCoerce)
import Crypto.Error
import Crypto.Number.Basic (numBits)
import Crypto.Random
@@ -135,6 +138,17 @@
hashUpdate (HashContextSSL sha1Ctx md5Ctx) b =
HashContextSSL (H.hashUpdate sha1Ctx b) (H.hashUpdate md5Ctx b)
+-- | Copies the context into a new allocation.
+hashCopy :: HashContext -> IO HashContext
+hashCopy (HashContext (ContextSimple h)) = HashContext . ContextSimple <$> copyContext h
+hashCopy (HashContextSSL sha1Ctx md5Ctx) = HashContextSSL <$> copyContext sha1Ctx <*> copyContext md5Ctx
+
+-- H.Context is a newtype over Bytes, with the constructor in a hidden crypton module
+copyContext :: H.Context alg -> IO (H.Context alg)
+copyContext h = do
+ b <- BA.copy h (\_ -> return ()) :: IO BA.Bytes
+ return $! unsafeCoerce b
+
hashUpdateSSL :: HashCtx
-> (B.ByteString,B.ByteString) -- ^ (for the md5 context, for the sha1 context)
-> HashCtx
diff -ruN -x 'dist*' tls-1.9.0-orig/Network/TLS/Handshake/Compact.hs tls-fork/Network/TLS/Handshake/Compact.hs
--- tls-1.9.0-orig/Network/TLS/Handshake/Compact.hs 1970-01-01 00:00:00.000000000 +0000
+++ tls-fork/Network/TLS/Handshake/Compact.hs 2026-09-30 18:45:10.137984909 +0000
@@ -0,0 +1,111 @@
+{-# LANGUAGE OverloadedStrings #-}
+{-# OPTIONS_HADDOCK hide #-}
+module Network.TLS.Handshake.Compact
+ ( compactContext
+ ) where
+
+import Control.Concurrent.MVar
+import Control.Exception (evaluate)
+import Crypto.Random (drgNew)
+import qualified Data.ByteString as B
+import Data.IORef
+
+import Network.TLS.Cipher
+import Network.TLS.Context.Internal
+import Network.TLS.Crypto
+import Network.TLS.Extension (Cookie (..))
+import Network.TLS.Handshake.State
+import Network.TLS.Imports
+import Network.TLS.KeySchedule (hkdfExpandLabel)
+import Network.TLS.Record.State
+import Network.TLS.RNG
+import Network.TLS.State
+import Network.TLS.Struct
+import Network.TLS.Types
+
+-- | Re-allocates the state a context keeps after the handshake, next to each other.
+-- Each small pinned value created during the handshake shares a 4K block with short-lived
+-- crypto temporaries, and keeps the whole block alive, with every dead ScrubbedBytes in it.
+compactContext :: Context -> IO ()
+compactContext ctx = do
+ tx <- readMVar $ ctxTxState ctx
+ rx <- readMVar $ ctxRxState ctx
+ st <- readMVar $ ctxState ctx
+ hst <- readMVar $ ctxHandshake ctx
+ finished <- readIORef $ ctxFinished ctx
+ peerFinished <- readIORef $ ctxPeerFinished ctx
+ -- key derivation and reseeding allocate temporaries, so they precede the copies
+ txKey <- trafficKey tx
+ rxKey <- trafficKey rx
+ rng <- evaluate $ StateRNG $ fst $ withTLSRNG (stRandomGen st) drgNew
+ tx' <- copyRecordState BulkEncrypt txKey tx
+ rx' <- copyRecordState BulkDecrypt rxKey rx
+ st' <- copyTLSState rng st
+ hst' <- traverse copyHandshakeState hst
+ finished' <- traverse copyBytes finished
+ peerFinished' <- traverse copyBytes peerFinished
+ modifyMVar_ (ctxTxState ctx) $ \_ -> return tx'
+ modifyMVar_ (ctxRxState ctx) $ \_ -> return rx'
+ modifyMVar_ (ctxState ctx) $ \_ -> return st'
+ modifyMVar_ (ctxHandshake ctx) $ \_ -> return hst'
+ writeIORef (ctxFinished ctx) finished'
+ writeIORef (ctxPeerFinished ctx) peerFinished'
+
+copyBytes :: ByteString -> IO ByteString
+copyBytes = evaluate . B.copy
+
+-- TLS 1.3 record keys are derived from the traffic secret kept in the record state;
+-- earlier versions keep the bulk key only inside the cipher state.
+trafficKey :: RecordState -> IO (Maybe ByteString)
+trafficKey RecordState{stCipher = Just cipher, stCryptLevel = CryptApplicationSecret, stCryptState = cst} =
+ Just <$> evaluate (hkdfExpandLabel (cipherHash cipher) (cstMacSecret cst) "key" "" (bulkKeySize $ cipherBulk cipher))
+trafficKey _ = return Nothing
+
+copyRecordState :: BulkDirection -> Maybe ByteString -> RecordState -> IO RecordState
+copyRecordState dir key_ rs@RecordState{stCryptState = cst} = do
+ iv <- copyBytes $ cstIV cst
+ secret <- copyBytes $ cstMacSecret cst
+ bulkState <- case (key_, stCipher rs) of
+ (Just key, Just cipher) -> bulkInit (cipherBulk cipher) dir <$> copyBytes key
+ _ -> return $ cstKey cst
+ evaluate rs{stCryptState = CryptState{cstKey = bulkState, cstIV = iv, cstMacSecret = secret}}
+
+copyTLSState :: StateRNG -> TLSState -> IO TLSState
+copyTLSState rng st = do
+ session <- case stSession st of
+ Session sid -> Session <$> traverse copyBytes sid
+ clientVerified <- copyBytes $ stClientVerifiedData st
+ serverVerified <- copyBytes $ stServerVerifiedData st
+ proto <- traverse copyBytes $ stNegotiatedProtocol st
+ alpn <- traverse (traverse copyBytes) $ stClientALPNSuggest st
+ cookie <- traverse (\(Cookie c) -> Cookie <$> copyBytes c) $ stTLS13Cookie st
+ exporter <- traverse copyBytes $ stExporterMasterSecret st
+ evaluate
+ st
+ { stSession = session
+ , stClientVerifiedData = clientVerified
+ , stServerVerifiedData = serverVerified
+ , stNegotiatedProtocol = proto
+ , stClientALPNSuggest = alpn
+ , stTLS13Cookie = cookie
+ , stExporterMasterSecret = exporter
+ , stRandomGen = rng
+ }
+
+copyHandshakeState :: HandshakeState -> IO HandshakeState
+copyHandshakeState hst = do
+ clientRandom <- ClientRandom <$> copyBytes (unClientRandom $ hstClientRandom hst)
+ serverRandom <- traverse (fmap ServerRandom . copyBytes . unServerRandom) $ hstServerRandom hst
+ mainSecret <- traverse copyBytes $ hstMasterSecret hst
+ digest <- case hstHandshakeDigest hst of
+ HandshakeMessages msgs -> HandshakeMessages <$> traverse copyBytes msgs
+ HandshakeDigestContext h -> HandshakeDigestContext <$> hashCopy h
+ resumption <- traverse (\(BaseSecret s) -> BaseSecret <$> copyBytes s) $ hstTLS13ResumptionSecret hst
+ evaluate
+ hst
+ { hstClientRandom = clientRandom
+ , hstServerRandom = serverRandom
+ , hstMasterSecret = mainSecret
+ , hstHandshakeDigest = digest
+ , hstTLS13ResumptionSecret = resumption
+ }
diff -ruN -x 'dist*' tls-1.9.0-orig/Network/TLS/Handshake.hs tls-fork/Network/TLS/Handshake.hs
--- tls-1.9.0-orig/Network/TLS/Handshake.hs 2026-09-30 18:44:40.083878116 +0000
+++ tls-fork/Network/TLS/Handshake.hs 2026-09-30 18:45:14.390780619 +0000
@@ -18,6 +18,7 @@
import Network.TLS.Struct
import Network.TLS.Handshake.Common
+import Network.TLS.Handshake.Compact
import Network.TLS.Handshake.Client
import Network.TLS.Handshake.Server
@@ -27,7 +28,7 @@
-- This is to be called at the beginning of a connection, and during renegotiation
handshake :: MonadIO m => Context -> m ()
handshake ctx =
- liftIO $ withRWLock ctx $ handleException ctx (ctxDoHandshake ctx ctx)
+ liftIO $ withRWLock ctx $ handleException ctx (ctxDoHandshake ctx ctx) >> compactContext ctx
-- Handshake when requested by the remote end
-- This is called automatically by 'recvData', in a context where the read lock
diff -ruN -x 'dist*' tls-1.9.0-orig/tls.cabal tls-fork/tls.cabal
--- tls-1.9.0-orig/tls.cabal 2026-09-30 18:44:40.090442445 +0000
+++ tls-fork/tls.cabal 2026-09-30 19:12:16.099341583 +0000
@@ -85,6 +85,7 @@
Network.TLS.Handshake.Signature
Network.TLS.Handshake.State
Network.TLS.Handshake.State13
+ Network.TLS.Handshake.Compact
Network.TLS.Hooks
Network.TLS.IO
Network.TLS.Imports