diff --git a/rfcs/2026-03-20-smp-agent-web/2026-05-22-agent.md b/rfcs/2026-03-20-smp-agent-web/2026-05-22-agent.md new file mode 100644 index 000000000..f19b1541d --- /dev/null +++ b/rfcs/2026-03-20-smp-agent-web/2026-05-22-agent.md @@ -0,0 +1,195 @@ +# Agent for Browser: Transpilation Breakdown + +**Parent**: [SMP Client MVP](./2026-05-20-client-mvp.md) +**Depends on**: SMP Client (complete, 96 tests), encoding/encryption spike (complete) + +## Rule + +Every TypeScript function is a faithful transpilation of a specific Haskell function. Same name, same steps, same call chain. No inferences. + +## Scope + +The web widget JOINS connections (never creates addresses). It sends and receives messages. It handles the connection handshake. It does NOT create invitations, manage notifications, transfer files, or do remote control. + +## Architecture difference from Haskell + +Haskell agent uses SQLite + multiple background threads (subscriber, delivery workers, cleanup manager, NTF supervisor). Browser agent uses IndexedDB + event-driven architecture (WebSocket onmessage, Promises, no threads). + +The protocol logic is identical. The concurrency model differs. The store interface differs. The transpilation focuses on the protocol logic. + +## Breakdown into testable pieces + +### Piece 1: Agent protocol types (Agent/Protocol.hs) + +Already partially done (AgentMsgEnvelope, AgentMessage, APrivHeader, AMessage). What's missing for the handshake: + +| Type | Haskell location | What's needed | +|------|-----------------|---------------| +| `SMPQueueInfo` | `Agent/Protocol.hs:1310-1327` | Binary encode/decode — version-dependent, complex | +| `SMPQueueUri` | `Agent/Protocol.hs:1344-1431` | Binary encode/decode + string encode/decode | +| `SMPQueueAddress` | `Agent/Protocol.hs:1350-1356` | `{smpServer, senderId, dhPublicKey, queueMode}` | +| `ConnectionRequestUri` | `Agent/Protocol.hs:1436-1441` | Binary encode/decode: `CRInvitationUri` + `CRContactUri` | +| `ConnReqUriData` | `Agent/Protocol.hs:1728-1734` | Binary encode/decode: `{crAgentVRange, crSmpQueues, crClientData}` | +| `SMPConfirmation` | `Agent/Protocol.hs:798-810` | Not wire-encoded — internal data structure for confirmation handling | +| `E2ERatchetParams` | `Crypto/Ratchet.hs:223-241` | Already have encode/decode in ratchet.ts — need to verify completeness | + +**Test**: each encode/decode function tested byte-for-byte against Haskell via callNode. + +### Piece 2: Connection handshake — joinConnection (Agent.hs) + +The join flow, transpiled step by step: + +``` +joinConnection (Agent.hs:~1200-1300) + 1. Parse ConnectionRequestUri (already have URI parsing) + 2. Create RcvQueue on a selected server (newRcvQueue — Agent/Client.hs:1373) + - Generate X25519 DH keypair for queue + - Generate X25519/Ed25519 auth keypair + - Call createSMPQueue on SMP client + - Get back rcvId, sndId, srvDhKey + 3. Store connection + queue in database + 4. Generate X448 E2E ratchet params (generateRcvE2EParams — already have) + 5. Build ConnInfo (profile data) + 6. Encrypt ConnInfo with ratchet → encConnInfo + 7. Build AgentConfirmation envelope + 8. Wrap in ClientMessage + per-queue E2E encrypt → ClientMsgEnvelope + 9. Send via SMP SEND to the contact address queue + 10. Subscribe to own receive queue (SUB) + 11. Return connection ID +``` + +Each step is independently testable. The full flow is an integration test. + +**Key Haskell functions to transpile:** + +| Function | File:lines | What it does | +|----------|-----------|--------------| +| `joinConnection` | `Agent.hs:~1200` | Top-level join | +| `joinConn` | `Agent.hs:~1230` | Internal join logic | +| `newRcvQueue` | `Agent/Client.hs:1373-1420` | Create queue on server | +| `sendConfirmation` | `Agent/Client.hs:1788-1794` | Encrypt+send confirmation | +| `sendInvitation` | `Agent/Client.hs:1796-1806` | Encrypt+send invitation | +| `mkAgentConfirmation` | `Agent.hs:~3700` | Build confirmation envelope | +| `agentCbEncrypt` | `Agent/Client.hs:2074-2082` | Per-queue E2E encrypt (already have) | + +### Piece 3: Message processing — subscriber (Agent.hs) + +Incoming message handling: + +``` +subscriber (Agent.hs:2912-2919) + → reads from msgQ (populated by SMP client's onMessage callback) + → processSMPTransmissions (Agent.hs:2997-3297) + → for each transmission: + → STEvent (server push MSG): + → decryptClientMessage (per-queue E2E decrypt) + → parse AgentMsgEnvelope + → for AgentMsgEnvelope 'M': + → agentRatchetDecrypt (double ratchet decrypt) + → parse AgentMessage + → dispatch on AMessage type: + → HELLO: complete handshake + → A_MSG body: deliver to user + → A_RCVD: delivery receipt + → QADD/QKEY/QUSE/QTEST: queue switching (skip for MVP) + → EREADY: ratchet sync (skip for MVP) +``` + +**Key Haskell functions to transpile:** + +| Function | File:lines | What it does | +|----------|-----------|--------------| +| `processSMPTransmissions` | `Agent.hs:2997-3297` | Top-level message dispatcher | +| `decryptClientMessage` | `Agent.hs:3282-3293` | Per-queue E2E decrypt + parse envelope | +| `agentRatchetDecrypt` | `Agent.hs:3757-3767` | Ratchet decrypt + store update | +| `helloMsg` | `Agent.hs:~3400` | Process HELLO (complete handshake) | +| `smpConfirmation` | `Agent.hs:~3500` | Process received confirmation | +| `smpInvitation` | `Agent.hs:~3600` | Process received invitation | + +### Piece 4: Message sending — sendMessage (Agent.hs) + +``` +sendMessage (Agent.hs:~1500) + → enqueueMessageB + → agentRatchetEncryptHeader (get ratchet encrypt key) + → rcEncryptMsg (encrypt message body) + → store encrypted message in DB + → createSndMsgDelivery (link to queue) + → delivery worker sends via SMP SEND +``` + +**Key Haskell functions to transpile:** + +| Function | File:lines | What it does | +|----------|-----------|--------------| +| `sendMessage` | `Agent.hs:~1500` | Top-level send | +| `enqueueMessageB` | `Agent.hs:~2020-2060` | Encode + encrypt + store | +| `agentRatchetEncrypt` | `Agent.hs:3742-3746` | Ratchet encrypt | +| `agentRatchetEncryptHeader` | `Agent.hs:3748-3754` | Ratchet encrypt header | +| `encodeAgentMsgStr` | `Agent.hs:2050-2054` | Encode AgentMessage to bytes | +| `runSmpQueueMsgDelivery` | `Agent.hs:2092-2220` | Delivery worker — read from DB, send via SMP | + +### Piece 5: Message acknowledgment (Agent.hs) + +``` +ackMessage (Agent.hs:~1550) + → withStore: mark message as acknowledged + → send ACK to SMP server + → optionally send delivery receipt (A_RCVD) +``` + +### Piece 6: Store interface (IndexedDB) + +The agent uses ~50 distinct store operations. For the web widget MVP, we need: + +| Store operation | What it does | Used by | +|----------------|--------------|---------| +| `createConnection` | Create connection record | joinConnection | +| `getConn` | Get connection by ID | all operations | +| `updateRcvIds` | Increment receive IDs | message processing | +| `createRcvMsg` | Store received message | message processing | +| `getRatchetForUpdate` | Get ratchet state for modify | encrypt/decrypt | +| `updateRatchet` | Store updated ratchet | encrypt/decrypt | +| `getSkippedMsgKeys` | Get skipped message keys | ratchet decrypt | +| `createSndMsg` | Store sent message | sendMessage | +| `createSndMsgDelivery` | Link message to queue | sendMessage | +| `getPendingQueueMsg` | Get next message to send | delivery worker | +| `updateSndMsgStatus` | Update delivery status | delivery worker | +| `deleteMsg` | Delete after ACK | ackMessage | + +## Implementation order + +1. **Piece 1**: Agent protocol types — `SMPQueueInfo`, `SMPQueueUri`, `ConnectionRequestUri`, `ConnReqUriData` encode/decode. Test each against Haskell. +2. **Piece 6**: Store interface — define TypeScript interface matching the needed operations. Implement with in-memory Map first (for testing), IndexedDB later. +3. **Piece 4**: Message sending — `sendMessage` + `agentRatchetEncrypt` + delivery. Test: encrypt in TS, decrypt in HS. +4. **Piece 3**: Message processing — `processSMPTransmissions` + `agentRatchetDecrypt`. Test: encrypt in HS, decrypt in TS. +5. **Piece 2**: Connection handshake — `joinConnection`. Test: TS joins, HS accepts, messages flow. +6. **Piece 5**: Message acknowledgment — `ackMessage`. Test: full roundtrip with ack. + +Each piece is independently testable. Piece 1 uses callNode (no server). Pieces 3-5 use the SMP client REPL + real server. Piece 2 is the integration test that ties everything together. + +## What to skip + +| Feature | Why skip | Haskell functions | +|---------|----------|-------------------| +| Queue switching | Additive, not needed for MVP | `switchConnection`, QADD/QKEY/QUSE/QTEST handling | +| Ratchet sync | MVP shows error, suggests reconnecting | `synchronizeRatchet`, EREADY handling | +| Notifications | No webpush yet | All NTF functions | +| File transfer | Separate protocol | All XFTP functions | +| Remote control | Not in scope | All RC functions | +| Client notices | Server admin feature | `processClientNotices` | +| Cleanup manager | Can do manual cleanup | `cleanupManager` | +| Server management | Configured at init | `setProtocolServers`, `testProtocolServer` | +| Connection creation | Widget only joins | `createConnection`, short links | +| Delivery receipts | Can add later | A_RCVD handling, receipt sending | +| Multiple receive queues | Single queue per connection for MVP | Queue replacement logic | + +## Files + +| File | Action | +|------|--------| +| `smp-web/src/agent/protocol.ts` | Extend with SMPQueueInfo, ConnectionRequestUri, etc. | +| `smp-web/src/agent/store.ts` | New — store interface + in-memory implementation | +| `smp-web/src/agent/agent.ts` | New — agent logic: join, send, receive, ack | +| `smp-web/tests/agent-repl.ts` | New — agent REPL for testing (or extend client-repl) | +| `tests/SMPWebTests.hs` | Agent tests | diff --git a/rfcs/2026-03-20-smp-agent-web/2026-05-23-agent-api-inventory.md b/rfcs/2026-03-20-smp-agent-web/2026-05-23-agent-api-inventory.md new file mode 100644 index 000000000..d0c303b01 --- /dev/null +++ b/rfcs/2026-03-20-smp-agent-web/2026-05-23-agent-api-inventory.md @@ -0,0 +1,260 @@ +# Agent API Inventory for Web Widget + +Every exported function from `Agent.hs`, classified as MVP / post-MVP / skip, with reasoning. + +## Context + +The web widget: +- Joins existing connections via addresses hardcoded in the widget or sent as simplex links +- Sends and receives messages +- Does NOT create addresses, invitation links, or short links +- Does NOT transfer files (post-MVP) +- Does NOT manage notifications (post-MVP) +- Does NOT rotate queues +- Must accept ratchet re-sync initiated by the other side (but does not initiate) + +## Lifecycle + +| Function | MVP? | Reason | +|----------|------|--------| +| `getSMPAgentClient` | YES | Initialize agent — required | +| `getSMPAgentClient_` | NO | Variant with extra params, not needed | +| `disconnectAgentClient` | YES | Clean shutdown — required | +| `disposeAgentClient` | NO | Hard shutdown — disconnectAgentClient is sufficient | +| `resumeAgentClient` | NO | Widget doesn't suspend/resume — runs while page is open | +| `foregroundAgent` | NO | Mobile-only concept | +| `suspendAgent` | NO | Mobile-only concept | + +## User management + +| Function | MVP? | Reason | +|----------|------|--------| +| `createUser` | YES | Widget needs at least one user to own connections | +| `deleteUser` | NO | Widget doesn't delete users — page reload is cleanup | + +## Connection creation (widget never creates) + +| Function | MVP? | Reason | +|----------|------|--------| +| `createConnection` | NO | Widget joins, never creates | +| `createConnectionAsync` | NO | Same | +| `prepareConnectionLink` | NO | For creating links | +| `createConnectionForLink` | NO | For creating links | +| `setConnShortLink` | NO | For creating short links | +| `setConnShortLinkAsync` | NO | Same | +| `deleteConnShortLink` | NO | For managing short links | +| `getConnShortLink` | NO | For reading short links — widget uses hardcoded address | +| `getConnShortLinkAsync` | NO | Same | +| `getConnLinkPrivKey` | NO | For link management | +| `deleteLocalInvShortLink` | NO | For link cleanup | +| `changeConnectionUser` | NO | Widget has one user | + +## Joining connections + +| Function | MVP? | Reason | +|----------|------|--------| +| `prepareConnectionToJoin` | YES | Create connection record before join — prevents race with incoming confirmation | +| `joinConnection` | YES | Core function — join via address URI | +| `joinConnectionAsync` | NO | Async variant — widget can use sync joinConnection with await | +| `connRequestPQSupport` | YES | Determine PQ support from connection request — needed for join | + +## Handshake (accepting incoming) + +| Function | MVP? | Reason | +|----------|------|--------| +| `allowConnection` | YES | Allow connection after receiving CONF — part of handshake | +| `allowConnectionAsync` | NO | Async variant | +| `acceptContact` | YES | Accept contact request (for group join flow) | +| `acceptContactAsync` | NO | Async variant | +| `prepareConnectionToAccept` | YES | Prepare for accept — same race prevention as prepareConnectionToJoin | +| `rejectContact` | NO | Widget doesn't reject — it always accepts | + +## Subscription + +| Function | MVP? | Reason | +|----------|------|--------| +| `subscribeConnection` | YES | Subscribe to receive messages on one connection | +| `subscribeConnections` | YES | Batch subscribe — needed after page reload | +| `subscribeAllConnections` | YES | Subscribe everything for a user — simplest for widget | +| `resubscribeConnection` | YES | Resubscribe after network recovery | +| `resubscribeConnections` | YES | Batch resubscribe | +| `getConnectionMessages` | NO | Fetch stored messages — widget processes messages as they arrive | +| `getNotificationConns` | NO | Push notification management | +| `subscribeClientService` | NO | Service certificate management | + +## Messaging + +| Function | MVP? | Reason | +|----------|------|--------| +| `sendMessage` | YES | Send a message — core function | +| `sendMessages` | NO | Batch send — sendMessage is sufficient for MVP | +| `sendMessagesB` | NO | Batch send with error handling — optimization | +| `ackMessage` | YES | Acknowledge received message — required for protocol correctness | +| `ackMessageAsync` | NO | Async variant | + +## Queue management + +| Function | MVP? | Reason | +|----------|------|--------| +| `switchConnection` | NO | Queue rotation — not needed | +| `switchConnectionAsync` | NO | Same | +| `abortConnectionSwitch` | NO | Cancel rotation | +| `getConnectionQueueInfo` | NO | Debug info | +| `suspendConnection` | NO | Widget deletes or ignores, doesn't suspend | + +## Ratchet + +| Function | MVP? | Reason | +|----------|------|--------| +| `synchronizeRatchet` | NO | Widget doesn't initiate ratchet sync. But MUST handle incoming EREADY — that's in processSMPTransmissions, not a separate API call. | +| `getConnectionRatchetAdHash` | NO | Verification UI not in widget | + +## Connection cleanup + +| Function | MVP? | Reason | +|----------|------|--------| +| `deleteConnection` | YES | User may want to delete a conversation | +| `deleteConnectionAsync` | NO | Async variant | +| `deleteConnections` | NO | Batch delete — single delete sufficient | +| `deleteConnectionsAsync` | NO | Same | +| `getConnectionServers` | NO | Info only | +| `compareConnections` | NO | Database sync tool | +| `syncConnections` | NO | Database sync tool | + +## Server configuration + +| Function | MVP? | Reason | +|----------|------|--------| +| `setProtocolServers` | NO | Widget initialized with servers, doesn't change them | +| `checkUserServers` | NO | Admin function | +| `testProtocolServer` | NO | Admin function | +| `setNtfServers` | NO | No notifications in MVP | +| `setNetworkConfig` | NO | Widget uses default network config | +| `setUserNetworkInfo` | NO | Widget doesn't track network state changes | +| `reconnectAllServers` | NO | Widget handles reconnection via SMP client | +| `reconnectSMPServer` | NO | Same | + +## Notifications (all post-MVP) + +| Function | MVP? | Reason | +|----------|------|--------| +| `registerNtfToken` | NO | No webpush yet | +| `verifyNtfToken` | NO | Same | +| `checkNtfToken` | NO | Same | +| `deleteNtfToken` | NO | Same | +| `getNtfToken` | NO | Same | +| `getNtfTokenData` | NO | Same | +| `toggleConnectionNtfs` | NO | Same | + +## File transfer (all post-MVP) + +| Function | MVP? | Reason | +|----------|------|--------| +| `xftpStartWorkers` | NO | Post-MVP | +| `xftpStartSndWorkers` | NO | Same | +| `xftpReceiveFile` | NO | Same | +| `xftpDeleteRcvFile` | NO | Same | +| `xftpDeleteRcvFiles` | NO | Same | +| `xftpSendFile` | NO | Same | +| `xftpSendDescription` | NO | Same | +| `xftpDeleteSndFileInternal` | NO | Same | +| `xftpDeleteSndFilesInternal` | NO | Same | +| `xftpDeleteSndFileRemote` | NO | Same | +| `xftpDeleteSndFilesRemote` | NO | Same | + +## Remote control (all skip) + +| Function | MVP? | Reason | +|----------|------|--------| +| `rcNewHostPairing` | NO | Not in scope | +| `rcConnectHost` | NO | Same | +| `rcConnectCtrl` | NO | Same | +| `rcDiscoverCtrl` | NO | Same | + +## Debug/stats (all skip) + +| Function | MVP? | Reason | +|----------|------|--------| +| `getAgentSubsTotal` | NO | Debug | +| `getAgentServersSummary` | NO | Debug | +| `resetAgentServersStats` | NO | Debug | +| `execAgentStoreSQL` | NO | Debug | +| `getAgentMigrations` | NO | Debug | +| `debugAgentLocks` | NO | Debug | +| `getAgentSubscriptions` | NO | Debug | +| `logConnection` | NO | Debug | +| `withAgentEnv` | NO | Test utility | + +## Summary + +### Corrections after reviewing simplex-chat-2 Subscriber.hs + Commands.hs + +The widget handles business chats (groups). Group flows trigger agent calls the widget doesn't initiate but must support: + +- Member introductions create connections asynchronously → `createConnectionAsync` +- Members join via introductions → `joinConnectionAsync`, `prepareConnectionToJoin` +- Members leave/deleted → `deleteConnectionAsync`, `deleteConnectionsAsync` +- Group messages go to all members → `sendMessages` +- All message acks use async variant → `ackMessageAsync` +- Accepting group invitations → `allowConnectionAsync` +- `acceptContact` — used in contact request acceptance flow (APIAcceptContactRequest in Commands.hs) +- `toggleConnectionNtfs` — used when CON received for group member (Subscriber.hs:850) + +### MVP functions (27 of 90): + +**Lifecycle:** `getSMPAgentClient`, `disconnectAgentClient` + +**User:** `createUser` + +**Join:** `prepareConnectionToJoin`, `joinConnection`, `joinConnectionAsync`, `connRequestPQSupport` + +**Handshake:** `allowConnection`, `allowConnectionAsync`, `acceptContact`, `acceptContactAsync`, `prepareConnectionToAccept` + +**Connection creation (for group member flows):** `createConnectionAsync`, `deleteConnectionAsync`, `deleteConnectionsAsync` + +**Subscribe:** `subscribeConnection`, `subscribeConnections`, `subscribeAllConnections`, `resubscribeConnection`, `resubscribeConnections` + +**Message:** `sendMessage`, `sendMessages`, `ackMessage`, `ackMessageAsync` + +**Cleanup:** `deleteConnection` + +**Notification toggle:** `toggleConnectionNtfs` + +**Internal (not exported but required):** `subscriber`/`processSMPTransmissions` (message processing), `runSmpQueueMsgDelivery` (delivery worker), all encryption/decryption functions, store operations for the above. + +### Message types to handle in processSMPTransmissions: + +| AMessage | Handle? | Reason | +|----------|---------|--------| +| `HELLO` | YES | Complete handshake | +| `A_MSG body` | YES | Deliver message to user | +| `A_RCVD receipts` | YES | Process delivery receipts (show checkmarks) | +| `A_QCONT addr` | NO | Queue continuation after quota — skip | +| `QADD qs` | NO | Queue rotation — skip | +| `QKEY qs` | NO | Queue rotation — skip | +| `QUSE qs` | NO | Queue rotation — skip | +| `QTEST qs` | NO | Queue rotation — skip | +| `EREADY msgId` | ACCEPT | Must handle incoming (don't initiate) — reset ratchet sync state | + +### AgentMsgEnvelope types to handle: + +| Variant | Handle? | Reason | +|---------|---------|--------| +| `AgentConfirmation` | YES | Handshake — received when peer confirms | +| `AgentMsgEnvelope` | YES | Normal encrypted messages | +| `AgentInvitation` | YES | Received when joining contact address | +| `AgentRatchetKey` | ACCEPT | Must handle incoming ratchet key — don't initiate | + +### Store operations needed: + +Based on the 18 MVP functions, the store needs (rough count): + +- Connection CRUD: ~8 operations +- Queue CRUD: ~6 operations +- Ratchet state: ~4 operations (get, update, skipped keys) +- Message storage: ~6 operations (create rcv/snd msg, update status, delete) +- Message delivery: ~4 operations (create delivery, get pending, update status) +- User: ~2 operations (create, get) +- Confirmation: ~3 operations (create, get, delete) + +**Estimated: ~33 store operations.** This is what determines whether SQLite WASM or IndexedDB direct is more practical. diff --git a/rfcs/2026-03-20-smp-agent-web/2026-05-24-agent-store.md b/rfcs/2026-03-20-smp-agent-web/2026-05-24-agent-store.md new file mode 100644 index 000000000..9026554b7 --- /dev/null +++ b/rfcs/2026-03-20-smp-agent-web/2026-05-24-agent-store.md @@ -0,0 +1,121 @@ +# Agent Store: IndexedDB Design + +**Parent**: [Agent Plan](./2026-05-22-agent.md) + +## Schema + +IndexedDB object stores, mapped from SQLite tables. Each store mirrors the Haskell schema from `agent_schema.sql`. + +### Object stores needed (16) + +``` +users + key: userId (autoincrement) + fields: deleted + +connections + key: connId (Uint8Array) + fields: connMode, lastInternalMsgId, lastInternalRcvMsgId, lastInternalSndMsgId, + lastExternalSndMsgId, lastRcvMsgHash, lastSndMsgHash, smpAgentVersion, + duplexHandshake, enableNtfs, deleted, userId, ratchetSyncState, pqSupport + +rcv_queues + key: [host, port, rcvId] (compound) + index: [connId], [host, port, sndId] + fields: connId, rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret, sndId, sndKey, + status, smpClientVersion, rcvQueueId, rcvPrimary, replaceRcvQueueId, queueMode, + serverKeyHash, lastBrokerTs + +snd_queues + key: [host, port, sndId] (compound) + index: [connId] + fields: connId, sndPrivateKey, e2eDhSecret, status, smpClientVersion, + sndPublicKey, e2ePubKey, sndQueueId, sndPrimary, queueMode, serverKeyHash + +messages + key: [connId, internalId] (compound) + index: [connId] + fields: internalTs, internalRcvId, internalSndId, msgType, msgBody, msgFlags, pqEncryption + +rcv_messages + key: [connId, internalRcvId] (compound) + index: [connId, internalId] + fields: internalId, externalSndId, brokerId, brokerTs, internalHash, + externalPrevSndHash, integrity, userAck, rcvQueueId, receiveAttempts + +snd_messages + key: [connId, internalSndId] (compound) + index: [connId, internalId] + fields: internalId, internalHash, previousMsgHash, retryIntSlow, retryIntFast, + rcptInternalId, rcptStatus, msgEncryptKey, paddedMsgLen, sndMessageBodyId + +snd_message_deliveries + key: sndMessageDeliveryId (autoincrement) + index: [connId, sndQueueId] + fields: connId, sndQueueId, internalId, failed + +snd_message_bodies + key: sndMessageBodyId (autoincrement) + fields: agentMsg + +conn_confirmations + key: confirmationId (Uint8Array) + index: [connId] + fields: connId, e2eSndPubKey, senderKey, ratchetState, senderConnInfo, + accepted, ownConnInfo, smpReplyQueues, smpClientVersion + +conn_invitations + key: invitationId (Uint8Array) + index: [contactConnId] + fields: contactConnId, crInvitation, recipientConnInfo, accepted, ownConnInfo + +ratchets + key: connId (Uint8Array) + fields: x3dhPrivKey1, x3dhPrivKey2, ratchetState, e2eVersion, + x3dhPubKey1, x3dhPubKey2, pqPrivKem, pqPubKem + +skipped_messages + key: skippedMessageId (autoincrement) + index: [connId] + fields: connId, headerKey, msgN, msgKey + +servers + key: [host, port] (compound) + fields: keyHash + +commands + key: commandId (autoincrement) + index: [connId], [host, port] + fields: connId, host, port, corrId, commandTag, command, agentVersion, serverKeyHash, failed + +encrypted_rcv_message_hashes + key: id (autoincrement) + index: [connId, hash] + fields: connId, hash, createdAt +``` + +## Interface + +TypeScript interface matching the ~60 store operations. Each method maps to a specific Haskell function in `AgentStore.hs`. + +The interface will be defined in `src/agent/store.ts`. Implementation in `src/agent/store-idb.ts` (IndexedDB). + +## Implementation approach + +1. Define the TypeScript interface first — every method name matches the Haskell function name +2. Implement with IndexedDB transactions +3. Test each operation in isolation before wiring to agent + +IndexedDB transactions map to SQLite transactions — both are ACID within a single store/table. Cross-store atomicity in IndexedDB requires putting multiple stores in one transaction, which is supported. + +## Key differences from SQLite + +1. **No SQL joins** — denormalize where needed, or do application-level joins +2. **No AUTO INCREMENT guaranteed ordering** — use explicit counters +3. **Blob keys** — IndexedDB supports ArrayBuffer keys natively +4. **Compound keys** — IndexedDB supports array keys: `[host, port, rcvId]` +5. **Indexes** — must be declared upfront in `onupgradeneeded` + +## Testing + +Each store operation tested by: write data, read it back, verify it matches. No server needed — pure store tests using `fake-indexeddb` in Node.js. diff --git a/smp-web/package.json b/smp-web/package.json index ab9ff7c1e..9f7faffb6 100644 --- a/smp-web/package.json +++ b/smp-web/package.json @@ -29,6 +29,7 @@ "devDependencies": { "@types/node": "^25.5.0", "@types/ws": "^8.18.1", + "fake-indexeddb": "^6.2.5", "typescript": "^5.4.0", "ws": "^8.0.0" } diff --git a/smp-web/src/agent/protocol.ts b/smp-web/src/agent/protocol.ts index 36b6e8f4a..4e7c00064 100644 --- a/smp-web/src/agent/protocol.ts +++ b/smp-web/src/agent/protocol.ts @@ -3,8 +3,15 @@ import {base64urlDecode} from "@simplex-chat/xftp-web/dist/protocol/description.js" import { - Decoder, decodeBytes, decodeLarge, decodeWord16, decodeBool, + Decoder, concatBytes, + encodeBytes, decodeBytes, + encodeLarge, decodeLarge, + encodeWord16, decodeWord16, + encodeBool, decodeBool, + encodeMaybe, decodeMaybe, + encodeNonEmpty, decodeNonEmpty, } from "@simplex-chat/xftp-web/dist/protocol/encoding.js" +import {encodeProtocolServer} from "../protocol.js" // -- Short link types (Agent/Protocol.hs:1462-1470) @@ -164,6 +171,196 @@ export function decodeFixedLinkData(d: Decoder): FixedLinkData { return {agentVRange: {min, max}, rootKey, rest} } +// -- SMPQueueAddress (Agent/Protocol.hs:1350-1356) + +export interface SMPQueueAddress { + smpServer: {hosts: string[], port: string, keyHash: Uint8Array} // ProtocolServer + senderId: Uint8Array // EntityId (ByteString) + dhPublicKey: Uint8Array // PublicKeyX25519 (DER-encoded ByteString) + queueMode: string | null // Maybe QueueMode: 'M' = Messaging, 'C' = Contact +} + +// -- SMPQueueInfo (Agent/Protocol.hs:1310-1327) +// Version-dependent encoding + +// SMP client version constants (Protocol.hs:281-294) +const initialSMPClientVersion = 1 +const sndAuthKeySMPClientVersion = 3 +const shortLinksSMPClientVersion = 4 + +export interface SMPQueueInfo { + clientVersion: number // VersionSMPC (Word16) + queueAddress: SMPQueueAddress +} + +// smpEncode (Agent/Protocol.hs:1313-1321) +export function encodeSMPQueueInfo(q: SMPQueueInfo): Uint8Array { + const {clientVersion, queueAddress: {smpServer, senderId, dhPublicKey, queueMode}} = q + const addrEnc = concatBytes( + encodeWord16(clientVersion), + encodeProtocolServer(smpServer.hosts, smpServer.port, smpServer.keyHash), + encodeBytes(senderId), + encodeBytes(dhPublicKey), + ) + if (clientVersion >= shortLinksSMPClientVersion) { + // encode queueMode directly (Maybe QueueMode as char or empty) + const qmBytes = queueMode ? new Uint8Array([queueMode.charCodeAt(0)]) : new Uint8Array(0) + return concatBytes(addrEnc, qmBytes) + } + if (clientVersion >= sndAuthKeySMPClientVersion && senderCanSecure(queueMode)) { + return concatBytes(addrEnc, encodeBool(true)) + } + if (clientVersion > initialSMPClientVersion) { + return addrEnc + } + // v1 legacy — not supported by web widget + throw new Error("encodeSMPQueueInfo: legacy v1 not supported") +} + +// smpP (Agent/Protocol.hs:1322-1327) +export function decodeSMPQueueInfo(d: Decoder): SMPQueueInfo { + const clientVersion = decodeWord16(d) + // v1 legacy server encoding not supported + if (clientVersion <= initialSMPClientVersion) throw new Error("decodeSMPQueueInfo: legacy v1 not supported") + const smpServer = decodeProtocolServerTyped(d) + const senderId = decodeBytes(d) + const dhPublicKey = decodeBytes(d) + const queueMode = decodeQueueMode(d) + return {clientVersion, queueAddress: {smpServer, senderId, dhPublicKey, queueMode}} +} + +// -- SMPQueueUri (Agent/Protocol.hs:1347-1431) + +export interface SMPQueueUri { + clientVRange: {min: number, max: number} // VersionRangeSMPC + queueAddress: SMPQueueAddress +} + +// smpEncode (Agent/Protocol.hs:1417-1427) +export function encodeSMPQueueUri(q: SMPQueueUri): Uint8Array { + const {clientVRange: {min: minV, max: maxV}, queueAddress: {smpServer, senderId, dhPublicKey, queueMode}} = q + const addrEnc = concatBytes( + encodeWord16(minV), encodeWord16(maxV), + encodeProtocolServer(smpServer.hosts, smpServer.port, smpServer.keyHash), + encodeBytes(senderId), + encodeBytes(dhPublicKey), + ) + if (minV >= shortLinksSMPClientVersion) { + const qmBytes = queueMode ? new Uint8Array([queueMode.charCodeAt(0)]) : new Uint8Array(0) + return concatBytes(addrEnc, qmBytes) + } + if (minV >= sndAuthKeySMPClientVersion || (maxV >= sndAuthKeySMPClientVersion && senderCanSecure(queueMode))) { + return concatBytes(addrEnc, encodeBool(senderCanSecure(queueMode))) + } + return addrEnc +} + +// smpP (Agent/Protocol.hs:1428-1431) +export function decodeSMPQueueUri(d: Decoder): SMPQueueUri { + const min = decodeWord16(d) + const max = decodeWord16(d) + const smpServer = decodeProtocolServerTyped(d) + const senderId = decodeBytes(d) + const dhPublicKey = decodeBytes(d) + const queueMode = decodeQueueMode(d) + return {clientVRange: {min, max}, queueAddress: {smpServer, senderId, dhPublicKey, queueMode}} +} + +// -- ConnReqUriData (Agent/Protocol.hs:1728-1734, 1145-1158) + +export interface ConnReqUriData { + crAgentVRange: {min: number, max: number} // VersionRangeSMPA + crSmpQueues: SMPQueueUri[] // NonEmpty SMPQueueUri + crClientData: string | null // Maybe CRClientData (Text) +} + +// smpEncode (Agent/Protocol.hs:1145-1147) +export function encodeConnReqUriData(d: ConnReqUriData): Uint8Array { + const vr = concatBytes(encodeWord16(d.crAgentVRange.min), encodeWord16(d.crAgentVRange.max)) + const queues = encodeNonEmpty(encodeSMPQueueUri, d.crSmpQueues) + const clientData = d.crClientData !== null + ? concatBytes(new Uint8Array([0x31]), encodeLarge(new TextEncoder().encode(d.crClientData))) + : new Uint8Array([0x30]) + return concatBytes(vr, queues, clientData) +} + +// smpP (Agent/Protocol.hs:1148-1158) +export function decodeConnReqUriData(d: Decoder): ConnReqUriData { + const min = decodeWord16(d) + const max = decodeWord16(d) + const crSmpQueues = decodeNonEmpty(decodeSMPQueueUri, d) + // Patch queueMode: if Nothing, set to QMContact (Agent/Protocol.hs:1156-1158) + for (const q of crSmpQueues) { + if (q.queueAddress.queueMode === null) q.queueAddress.queueMode = "C" + } + const clientData = decodeMaybe((dd) => { + const large = decodeLarge(dd) + return new TextDecoder().decode(large) + }, d) + return {crAgentVRange: {min, max}, crSmpQueues, crClientData: clientData} +} + +// -- ConnectionRequestUri (Agent/Protocol.hs:1130-1143, 1436-1441) + +export type ConnectionRequestUri = + | {mode: "invitation", crData: ConnReqUriData, e2eParams: Uint8Array} // raw smpEncoded E2ERatchetParams + | {mode: "contact", crData: ConnReqUriData} + +// smpEncode (Agent/Protocol.hs:1130-1133) +export function encodeConnectionRequestUri(cr: ConnectionRequestUri): Uint8Array { + switch (cr.mode) { + case "invitation": + return concatBytes(new Uint8Array([0x49]), encodeConnReqUriData(cr.crData), cr.e2eParams) // 'I' + crData + e2eParams + case "contact": + return concatBytes(new Uint8Array([0x43]), encodeConnReqUriData(cr.crData)) // 'C' + crData + } +} + +// smpP (Agent/Protocol.hs:1140-1143) +export function decodeConnectionRequestUri(d: Decoder): ConnectionRequestUri { + const mode = d.anyByte() + if (mode === 0x49) { // 'I' Invitation + const crData = decodeConnReqUriData(d) + const e2eParams = d.takeAll() // E2ERatchetParams consumes rest + return {mode: "invitation", crData, e2eParams} + } + if (mode === 0x43) { // 'C' Contact + const crData = decodeConnReqUriData(d) + return {mode: "contact", crData} + } + throw new Error("decodeConnectionRequestUri: unknown mode 0x" + mode.toString(16)) +} + +// -- Helpers + +function senderCanSecure(queueMode: string | null): boolean { + return queueMode === "M" +} + +// queueModeP (Agent/Protocol.hs:1433-1434) +// Just <$> smpP <|> optional ((\case True -> QMMessaging; _ -> QMContact) <$> smpP) +function decodeQueueMode(d: Decoder): string | null { + if (d.remaining() === 0) return null + const b = d.anyByte() + if (b === 0x4D) return "M" // QMMessaging + if (b === 0x43) return "C" // QMContact + // Could be a Bool (sndSecure) for older versions — True='T'(0x54) → QMMessaging, False='F'(0x46) → QMContact + if (b === 0x54) return "M" // True → QMMessaging + if (b === 0x46) return null // False → no queueMode (not secured) + return null +} + +// Decode ProtocolServer into typed format with string hosts +function decodeProtocolServerTyped(d: Decoder): {hosts: string[], port: string, keyHash: Uint8Array} { + const hostCount = d.anyByte() + if (hostCount === 0) throw new Error("empty server host list") + const hosts: string[] = [] + for (let i = 0; i < hostCount; i++) hosts.push(new TextDecoder().decode(decodeBytes(d))) + const port = new TextDecoder().decode(decodeBytes(d)) + const keyHash = decodeBytes(d) + return {hosts, port, keyHash} +} + // -- Profile extraction export function parseProfile(userData: Uint8Array): unknown { diff --git a/smp-web/src/agent/store-idb.ts b/smp-web/src/agent/store-idb.ts new file mode 100644 index 000000000..c238950fe --- /dev/null +++ b/smp-web/src/agent/store-idb.ts @@ -0,0 +1,1604 @@ +// IndexedDB implementation of AgentStore. +// Each method is a faithful transpilation of SQL in AgentStore.hs. +// +// Transpilation approach: +// - Each SQL query is converted to equivalent IndexedDB operations +// - Field names match SQLite column names from agent_schema.sql +// - Transaction scoping matches the Haskell withStore patterns +// +// This file is a work in progress — methods are added as they are transpiled +// from the Haskell source. + +import type {AgentStore, ConnId, UserId} from "./store.js" + +const DB_NAME = "simplex-agent" +const DB_VERSION = 1 + +function toHex(bytes: Uint8Array): string { + return Array.from(bytes, b => b.toString(16).padStart(2, "0")).join("") +} + +function compareUint8Array(a: Uint8Array, b: Uint8Array): number { + const len = Math.min(a.length, b.length) + for (let i = 0; i < len; i++) { + if (a[i] !== b[i]) return a[i] - b[i] + } + return a.length - b.length +} + +// -- Database opening + schema creation (from agent_schema.sql) + +export async function openAgentStore(): Promise { + const db = await openDB() + return createStore(db) +} + +function openDB(): Promise { + return new Promise((resolve, reject) => { + const req = indexedDB.open(DB_NAME, DB_VERSION) + req.onerror = () => reject(req.error) + req.onsuccess = () => resolve(req.result) + req.onupgradeneeded = () => createSchema(req.result) + }) +} + +// Schema mirrors agent_schema.sql tables needed for MVP +function createSchema(db: IDBDatabase): void { + // users — agent_schema.sql:266-270 + // CREATE TABLE users(user_id INTEGER PRIMARY KEY AUTOINCREMENT, deleted INTEGER DEFAULT 0) + db.createObjectStore("users", {keyPath: "user_id", autoIncrement: true}) + + // servers — agent_schema.sql:6-11 + // CREATE TABLE servers(host TEXT, port TEXT, key_hash BLOB, PRIMARY KEY(host, port)) + db.createObjectStore("servers", {keyPath: ["host", "port"]}) + + // connections — agent_schema.sql:12-31 + const conns = db.createObjectStore("connections", {keyPath: "conn_id"}) + conns.createIndex("user_id", "user_id") + + // rcv_queues — agent_schema.sql:32-70 + const rcvQ = db.createObjectStore("rcv_queues", {keyPath: ["host", "port", "rcv_id"]}) + rcvQ.createIndex("conn_id", "conn_id") + rcvQ.createIndex("conn_id_rcv_queue_id", ["conn_id", "rcv_queue_id"], {unique: true}) + + // snd_queues — agent_schema.sql:71-92 + const sndQ = db.createObjectStore("snd_queues", {keyPath: ["host", "port", "snd_id"]}) + sndQ.createIndex("conn_id", "conn_id") + sndQ.createIndex("conn_id_snd_queue_id", ["conn_id", "snd_queue_id"], {unique: true}) + + // messages — agent_schema.sql:93-109 + const msgs = db.createObjectStore("messages", {keyPath: ["conn_id", "internal_id"]}) + msgs.createIndex("conn_id", "conn_id") + msgs.createIndex("conn_id_internal_rcv_id", ["conn_id", "internal_rcv_id"]) + msgs.createIndex("conn_id_internal_snd_id", ["conn_id", "internal_snd_id"]) + + // rcv_messages — agent_schema.sql:110-126 + const rcvMsgs = db.createObjectStore("rcv_messages", {keyPath: ["conn_id", "internal_rcv_id"]}) + rcvMsgs.createIndex("conn_id_internal_id", ["conn_id", "internal_id"]) + rcvMsgs.createIndex("conn_id_broker_id", ["conn_id", "broker_id"]) + + // snd_messages — agent_schema.sql:127-143 + const sndMsgs = db.createObjectStore("snd_messages", {keyPath: ["conn_id", "internal_snd_id"]}) + sndMsgs.createIndex("conn_id_internal_id", ["conn_id", "internal_id"]) + + // snd_message_deliveries — agent_schema.sql:257-264 + const sndDel = db.createObjectStore("snd_message_deliveries", {keyPath: "snd_message_delivery_id", autoIncrement: true}) + sndDel.createIndex("conn_id_snd_queue_id", ["conn_id", "snd_queue_id"]) + + // snd_message_bodies — agent_schema.sql:428-431 + db.createObjectStore("snd_message_bodies", {keyPath: "snd_message_body_id", autoIncrement: true}) + + // conn_confirmations — agent_schema.sql:144-157 + const confs = db.createObjectStore("conn_confirmations", {keyPath: "confirmation_id"}) + confs.createIndex("conn_id", "conn_id") + + // conn_invitations — agent_schema.sql:158-166 + const invs = db.createObjectStore("conn_invitations", {keyPath: "invitation_id"}) + invs.createIndex("contact_conn_id", "contact_conn_id") + + // ratchets — agent_schema.sql:167-181 + db.createObjectStore("ratchets", {keyPath: "conn_id"}) + + // skipped_messages — agent_schema.sql:182-189 + const skip = db.createObjectStore("skipped_messages", {keyPath: "skipped_message_id", autoIncrement: true}) + skip.createIndex("conn_id", "conn_id") + + // commands — agent_schema.sql:242-256 + const cmds = db.createObjectStore("commands", {keyPath: "command_id", autoIncrement: true}) + cmds.createIndex("conn_id", "conn_id") + cmds.createIndex("host_port", ["host", "port"]) + + // encrypted_rcv_message_hashes — agent_schema.sql:397-403 + const hashes = db.createObjectStore("encrypted_rcv_message_hashes", {keyPath: "encrypted_rcv_message_hash_id", autoIncrement: true}) + hashes.createIndex("conn_id_hash", ["conn_id", "hash"]) +} + +// -- Helpers + +function idbReq(r: IDBRequest): Promise { + return new Promise((resolve, reject) => { + r.onsuccess = () => resolve(r.result) + r.onerror = () => reject(r.error) + }) +} + +function eqBytes(a: Uint8Array, b: Uint8Array): boolean { + if (a.length !== b.length) return false + for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) return false + return true +} + +// IndexedDB index.getAll() with Uint8Array keys doesn't work in fake-indexeddb polyfill. +// These helpers scan all records and filter by field values, handling Uint8Array comparison. +function fieldEq(a: any, b: any): boolean { + if (a instanceof Uint8Array && b instanceof Uint8Array) return eqBytes(a, b) + return a === b +} + +async function allByIndex(store: IDBObjectStore, fields: Record): Promise { + const all = await idbReq(store.getAll()) + return all.filter(r => Object.entries(fields).every(([k, v]) => fieldEq(r[k], v))) +} + +function createStore(db: IDBDatabase): AgentStore { + // Transaction helper + function withTx(stores: string | string[], mode: IDBTransactionMode, fn: (tx: IDBTransaction) => Promise): Promise { + return new Promise((resolve, reject) => { + const tx = db.transaction(stores, mode) + let result: T + const p = fn(tx).then(r => { result = r }) + tx.oncomplete = () => p.then(() => resolve(result)) + tx.onerror = () => reject(tx.error) + tx.onabort = () => reject(tx.error) + }) + } + + return { + // ============================================================ + // Users — AgentStore.hs:341-397 + // ============================================================ + + // createUserRecord (AgentStore.hs:341-344) + // SQL: INSERT INTO users DEFAULT VALUES; SELECT last_insert_rowid() + async createUserRecord() { + return withTx("users", "readwrite", async (tx) => { + const store = tx.objectStore("users") + const id = await idbReq(store.add({deleted: 0})) + return id as number + }) + }, + + // getUserIds (AgentStore.hs:346-348) + // SQL: SELECT user_id FROM users WHERE deleted = 0 + async getUserIds() { + return withTx("users", "readonly", async (tx) => { + const all = await idbReq(tx.objectStore("users").getAll()) + return all.filter(u => u.deleted === 0).map(u => u.user_id) + }) + }, + + // deleteUserRecord (AgentStore.hs:355-358) + // SQL: DELETE FROM users WHERE user_id = ? + async deleteUserRecord(userId) { + return withTx("users", "readwrite", async (tx) => { + await idbReq(tx.objectStore("users").delete(userId)) + }) + }, + + // setUserDeleted (AgentStore.hs:360-365) + // SQL: UPDATE users SET deleted = 1 WHERE user_id = ? + // SQL: SELECT conn_id FROM connections WHERE user_id = ? + async setUserDeleted(userId) { + return withTx(["users", "connections"], "readwrite", async (tx) => { + const store = tx.objectStore("users") + const user = await idbReq(store.get(userId)) + if (user) { + user.deleted = 1 + await idbReq(store.put(user)) + } + }) + }, + + // ============================================================ + // Servers — AgentStore.hs:233-240 (approximate, createServer is simple) + // ============================================================ + + // createServer + // SQL: INSERT OR IGNORE INTO servers (host, port, key_hash) VALUES (?,?,?) + async createServer(host, port, keyHash) { + return withTx("servers", "readwrite", async (tx) => { + const store = tx.objectStore("servers") + const existing = await idbReq(store.get([host, port])) + if (!existing) await idbReq(store.add({host, port, key_hash: keyHash})) + }) + }, + + // ============================================================ + // Connections — AgentStore.hs:409-498 + // ============================================================ + + // createNewConn (AgentStore.hs:409-411) → calls createConnRecord (AgentStore.hs:442-450) + // SQL: INSERT INTO connections (user_id, conn_id, conn_mode, smp_agent_version, enable_ntfs, pq_support, duplex_handshake) VALUES (?,?,?,?,?,?,?) + async createNewConn(connData, connMode) { + return withTx("connections", "readwrite", async (tx) => { + await idbReq(tx.objectStore("connections").add({ + conn_id: connData.connId, + user_id: connData.userId, + conn_mode: connMode, + smp_agent_version: connData.smpAgentVersion, + enable_ntfs: connData.enableNtfs ? 1 : 0, + pq_support: connData.pqSupport ? 1 : 0, + duplex_handshake: 1, + deleted: 0, + ratchet_sync_state: "ok", + last_internal_msg_id: 0, + last_internal_rcv_msg_id: 0, + last_internal_snd_msg_id: 0, + last_external_snd_msg_id: 0, + last_rcv_msg_hash: new Uint8Array(0), + last_snd_msg_hash: new Uint8Array(0), + })) + return connData.connId + }) + }, + + // Placeholder for remaining methods — each will be transpiled from the actual SQL + // TODO: transpile remaining ~70 methods from AgentStore.hs + + // getConn (AgentStore.hs:2248-2280) + // SQL: SELECT ... FROM connections WHERE conn_id = ? AND deleted = 0 + // Then gets rcv_queues and snd_queues by conn_id + async getConn(connId) { + return withTx(["connections", "rcv_queues", "snd_queues", "servers"], "readonly", async (tx) => { + // getConnData: SELECT user_id, conn_id, conn_mode, smp_agent_version, enable_ntfs, last_external_snd_msg_id, deleted, ratchet_sync_state, pq_support FROM connections WHERE conn_id = ? AND deleted = 0 + const conn = await idbReq(tx.objectStore("connections").get(connId)) + if (!conn || conn.deleted !== 0) return null + // getRcvQueuesByConnId_: rcvQueueQuery WHERE q.conn_id = ? AND q.deleted = 0 + const rcvQueues = (await allByIndex(tx.objectStore("rcv_queues"), {conn_id: connId})).filter((q: any) => !q.deleted) + // getSndQueuesByConnId_: sndQueueQuery WHERE q.conn_id = ? + const sndQueues = await allByIndex(tx.objectStore("snd_queues"), {conn_id: connId}) + return {connData: conn, rcvQueues, sndQueues} + }) + }, + + // getRcvConn (AgentStore.hs:467-472) + // SQL: rcvQueueQuery WHERE q.host = ? AND q.port = ? AND q.rcv_id = ? AND q.deleted = 0 + // Then getConn for the connId + async getRcvConn(host, port, rcvId) { + return withTx(["rcv_queues", "connections", "snd_queues", "servers"], "readonly", async (tx) => { + const rq = await idbReq(tx.objectStore("rcv_queues").get([host, port, rcvId])) + if (!rq || rq.deleted) return null + const conn = await idbReq(tx.objectStore("connections").get(rq.conn_id)) + if (!conn || conn.deleted !== 0) return null + return {connData: conn, rcvQueue: rq} + }) + }, + + // getConnSubs (AgentStore.hs:2295-2297) — gets ConnData for multiple connIds + async getConnSubs(connIds) { + return withTx("connections", "readonly", async (tx) => { + const store = tx.objectStore("connections") + const result = new Map() + for (const cid of connIds) { + const conn = await idbReq(store.get(cid)) + if (conn && conn.deleted === 0) result.set(toHex(cid), conn) + } + return result + }) + }, + + // getConnsData (AgentStore.hs:2371) — same as getConnSubs for our purposes + async getConnsData(connIds) { + return this.getConnSubs(connIds) + }, + + // lockConnForUpdate (AgentStore.hs:2394-2399) + // Postgres: SELECT 1 FROM connections WHERE conn_id = ? FOR UPDATE + // SQLite/IndexedDB: no-op (single-threaded) + async lockConnForUpdate() { }, + + // setConnDeleted (AgentStore.hs:2405-2411) + // waitDelivery=true: UPDATE connections SET deleted_at_wait_delivery = ? WHERE conn_id = ? + // waitDelivery=false: UPDATE connections SET deleted = 1 WHERE conn_id = ? + async setConnDeleted(connId, waitDelivery) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { + if (waitDelivery) { + conn.deleted_at_wait_delivery = new Date().toISOString() + } else { + conn.deleted = 1 + } + await idbReq(store.put(conn)) + } + }) + }, + + // setConnUserId (AgentStore.hs:2413-2415) + // SQL: UPDATE connections SET user_id = ? WHERE conn_id = ? AND user_id = ? + async setConnUserId(oldUserId, connId, newUserId) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn && conn.user_id === oldUserId) { + conn.user_id = newUserId + await idbReq(store.put(conn)) + } + }) + }, + + // setConnAgentVersion (AgentStore.hs:2417-2419) + // SQL: UPDATE connections SET smp_agent_version = ? WHERE conn_id = ? + async setConnAgentVersion(connId, version) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { + conn.smp_agent_version = version + await idbReq(store.put(conn)) + } + }) + }, + + // setConnPQSupport (AgentStore.hs:2421-2423) + // SQL: UPDATE connections SET pq_support = ? WHERE conn_id = ? + async setConnPQSupport(connId, pqSupport) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { + conn.pq_support = pqSupport ? 1 : 0 + await idbReq(store.put(conn)) + } + }) + }, + + // setConnRatchetSync (AgentStore.hs:2436) + // SQL: UPDATE connections SET ratchet_sync_state = ? WHERE conn_id = ? + async setConnRatchetSync(connId, state) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { + conn.ratchet_sync_state = state + await idbReq(store.put(conn)) + } + }) + }, + + // updateNewConnJoin (AgentStore.hs:2425-2427) + // SQL: UPDATE connections SET smp_agent_version = ?, pq_support = ?, enable_ntfs = ? WHERE conn_id = ? + async updateNewConnJoin(connId, agentVersion, pqSupport, enableNtfs) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { + conn.smp_agent_version = agentVersion + conn.pq_support = pqSupport ? 1 : 0 + conn.enable_ntfs = enableNtfs ? 1 : 0 + await idbReq(store.put(conn)) + } + }) + }, + + // updateNewConnRcv (AgentStore.hs:414-422) + // Calls addConnRcvQueue_ which calls insertRcvQueue_ + async updateNewConnRcv(connId, rcvQueue, subMode) { + return this.addConnRcvQueue(connId, rcvQueue, subMode) + }, + + // getDeletedConnIds (AgentStore.hs:2429-2430) + // SQL: SELECT conn_id FROM connections WHERE deleted = 1 + async getDeletedConnIds() { + return withTx("connections", "readonly", async (tx) => { + const all = await idbReq(tx.objectStore("connections").getAll()) + return all.filter((c: any) => c.deleted === 1 && !c.deleted_at_wait_delivery).map((c: any) => c.conn_id) + }) + }, + + // getDeletedWaitingDeliveryConnIds (AgentStore.hs:2432-2434) + // SQL: SELECT conn_id FROM connections WHERE deleted_at_wait_delivery IS NOT NULL + async getDeletedWaitingDeliveryConnIds() { + return withTx("connections", "readonly", async (tx) => { + const all = await idbReq(tx.objectStore("connections").getAll()) + return all.filter((c: any) => c.deleted_at_wait_delivery != null).map((c: any) => c.conn_id) + }) + }, + + // getConnIds — SELECT conn_id FROM connections + async getConnIds() { + return withTx("connections", "readonly", async (tx) => { + return idbReq(tx.objectStore("connections").getAllKeys()) + }) + }, + + // addConnRcvQueue (AgentStore.hs:516-526 → insertRcvQueue_ at 2077-2099) + // SQL: INSERT INTO rcv_queues (host, port, rcv_id, conn_id, rcv_private_key, rcv_dh_secret, e2e_priv_key, e2e_dh_secret, snd_id, queue_mode, status, to_subscribe, rcv_queue_id, rcv_primary, ...) VALUES (...) + async addConnRcvQueue(connId, rcvQueue, subMode) { + return withTx(["rcv_queues", "servers"], "readwrite", async (tx) => { + // createServer (INSERT OR IGNORE) + const srvStore = tx.objectStore("servers") + if (!(await idbReq(srvStore.get([rcvQueue.host, rcvQueue.port])))) await idbReq(srvStore.add({host: rcvQueue.host, port: rcvQueue.port, key_hash: rcvQueue.serverKeyHash})) + // Determine rcv_queue_id: SELECT MAX(rcv_queue_id) FROM rcv_queues WHERE conn_id = ? + const existing = await allByIndex(tx.objectStore("rcv_queues"), {conn_id: connId}) + const maxId = existing.reduce((m: number, q: any) => Math.max(m, q.rcv_queue_id || 0), 0) + const qId = maxId + 1 + const toSubscribe = subMode === "SMOnlyCreate" ? 1 : 0 + await idbReq(tx.objectStore("rcv_queues").add({ + host: rcvQueue.host, port: rcvQueue.port, rcv_id: rcvQueue.rcvId, + conn_id: connId, rcv_private_key: rcvQueue.rcvPrivateKey, rcv_dh_secret: rcvQueue.rcvDhSecret, + e2e_priv_key: rcvQueue.e2ePrivKey, e2e_dh_secret: rcvQueue.e2eDhSecret, + snd_id: rcvQueue.sndId, queue_mode: rcvQueue.queueMode, status: rcvQueue.status, + to_subscribe: toSubscribe, rcv_queue_id: qId, rcv_primary: rcvQueue.primary ? 1 : 0, + replace_rcv_queue_id: rcvQueue.replaceRcvQueueId, smp_client_version: rcvQueue.smpClientVersion, + server_key_hash: rcvQueue.serverKeyHash, deleted: 0, + snd_key: rcvQueue.sndKey, last_broker_ts: rcvQueue.lastBrokerTs, + })) + return {...rcvQueue, connId, dbQueueId: qId} + }) + }, + + // addConnSndQueue (AgentStore.hs:528-538 → insertSndQueue_ at 2109-2140) + // SQL: INSERT INTO snd_queues (host, port, snd_id, queue_mode, conn_id, snd_private_key, e2e_pub_key, e2e_dh_secret, status, snd_queue_id, snd_primary, ...) VALUES (...) ON CONFLICT DO UPDATE + async addConnSndQueue(connId, sndQueue) { + return withTx(["snd_queues", "servers"], "readwrite", async (tx) => { + const srvStore2 = tx.objectStore("servers") + if (!(await idbReq(srvStore2.get([sndQueue.host, sndQueue.port])))) await idbReq(srvStore2.add({host: sndQueue.host, port: sndQueue.port, key_hash: sndQueue.serverKeyHash})) + const existing = await allByIndex(tx.objectStore("snd_queues"), {conn_id: connId}) + const maxId = existing.reduce((m: number, q: any) => Math.max(m, q.snd_queue_id || 0), 0) + const qId = maxId + 1 + // ON CONFLICT DO UPDATE → use put (upsert) + await idbReq(tx.objectStore("snd_queues").put({ + host: sndQueue.host, port: sndQueue.port, snd_id: sndQueue.sndId, + queue_mode: sndQueue.queueMode, conn_id: connId, + snd_private_key: sndQueue.sndPrivateKey, e2e_pub_key: sndQueue.e2ePubKey, + e2e_dh_secret: sndQueue.e2eDhSecret, status: sndQueue.status, + snd_queue_id: qId, snd_primary: sndQueue.primary ? 1 : 0, + replace_snd_queue_id: null, smp_client_version: sndQueue.smpClientVersion, + server_key_hash: sndQueue.serverKeyHash, snd_public_key: sndQueue.sndPublicKey, + })) + }) + }, + + // setRcvQueueStatus (AgentStore.hs:540-550) + // SQL: UPDATE rcv_queues SET status = ? WHERE host = ? AND port = ? AND rcv_id = ? + async setRcvQueueStatus(rcvQueue, status) { + return withTx("rcv_queues", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_queues") + const rq = await idbReq(store.get([rcvQueue.host, rcvQueue.port, rcvQueue.rcvId])) + if (rq) { rq.status = status; await idbReq(store.put(rq)) } + }) + }, + + // setSndQueueStatus (AgentStore.hs:588-598) + // SQL: UPDATE snd_queues SET status = ? WHERE host = ? AND port = ? AND snd_id = ? + async setSndQueueStatus(sndQueue, status) { + return withTx("snd_queues", "readwrite", async (tx) => { + const store = tx.objectStore("snd_queues") + const sq = await idbReq(store.get([sndQueue.host, sndQueue.port, sndQueue.sndId])) + if (sq) { sq.status = status; await idbReq(store.put(sq)) } + }) + }, + + // setRcvQueueConfirmedE2E (AgentStore.hs:575-586) + // SQL: UPDATE rcv_queues SET e2e_dh_secret = ?, status = 'confirmed', smp_client_version = ? WHERE host = ? AND port = ? AND rcv_id = ? + async setRcvQueueConfirmedE2E(rcvQueue, dhSecret, smpClientVersion) { + return withTx("rcv_queues", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_queues") + const rq = await idbReq(store.get([rcvQueue.host, rcvQueue.port, rcvQueue.rcvId])) + if (rq) { rq.e2e_dh_secret = dhSecret; rq.status = "confirmed"; rq.smp_client_version = smpClientVersion; await idbReq(store.put(rq)) } + }) + }, + + // setRcvQueuePrimary (AgentStore.hs:612-618) + // SQL1: UPDATE rcv_queues SET rcv_primary = 0 WHERE conn_id = ? + // SQL2: UPDATE rcv_queues SET rcv_primary = 1, replace_rcv_queue_id = NULL WHERE conn_id = ? AND rcv_queue_id = ? + async setRcvQueuePrimary(connId, rcvQueue) { + return withTx("rcv_queues", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_queues") + const all = await allByIndex(store, {conn_id: connId}) + for (const rq of all) { + if (rq.rcv_queue_id === rcvQueue.dbQueueId) { + rq.rcv_primary = 1; rq.replace_rcv_queue_id = null + } else { + rq.rcv_primary = 0 + } + await idbReq(store.put(rq)) + } + }) + }, + + // deleteConnRcvQueue (AgentStore.hs:632-634) + // SQL: DELETE FROM rcv_queues WHERE conn_id = ? AND rcv_queue_id = ? + async deleteConnRcvQueue(rcvQueue) { + return withTx("rcv_queues", "readwrite", async (tx) => { + await idbReq(tx.objectStore("rcv_queues").delete([rcvQueue.host, rcvQueue.port, rcvQueue.rcvId])) + }) + }, + + // deleteConnRecord (AgentStore.hs:452-453) + // SQL: DELETE FROM connections WHERE conn_id = ? + async deleteConnRecord(connId) { + return withTx("connections", "readwrite", async (tx) => { + await idbReq(tx.objectStore("connections").delete(connId)) + }) + }, + + // upgradeRcvConnToDuplex (AgentStore.hs:501-505) + // Adds snd queue to an existing rcv connection + async upgradeRcvConnToDuplex(connId, sndQueue) { + return this.addConnSndQueue(connId, sndQueue) + }, + + // upgradeSndConnToDuplex (AgentStore.hs:508-513) + // Adds rcv queue to an existing snd connection + async upgradeSndConnToDuplex(connId, rcvQueue, subMode) { + return this.addConnRcvQueue(connId, rcvQueue, subMode) + }, + + // getPrimaryRcvQueue (AgentStore.hs:641-643) + // Gets rcv queues by connId, returns first (primary, sorted) + async getPrimaryRcvQueue(connId) { + return withTx("rcv_queues", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("rcv_queues"), {conn_id: connId}) + const active = all.filter((q: any) => !q.deleted) + // Sort by primary first (primary=1 first) + active.sort((a: any, b: any) => (b.rcv_primary || 0) - (a.rcv_primary || 0)) + return active[0] ?? null + }) + }, + + // getRcvQueue (AgentStore.hs:645-648) + // SQL: rcvQueueQuery WHERE q.conn_id = ? AND q.host = ? AND q.port = ? AND q.rcv_id = ? AND q.deleted = 0 + async getRcvQueue(connId, host, port, rcvId) { + return withTx("rcv_queues", "readonly", async (tx) => { + const rq = await idbReq(tx.objectStore("rcv_queues").get([host, port, rcvId])) + if (!rq || rq.deleted || toHex(rq.conn_id) !== toHex(connId)) return null + return rq + }) + }, + + // getDeletedRcvQueue (AgentStore.hs:650-653) + // SQL: same but q.deleted = 1 + async getDeletedRcvQueue(connId, host, port, rcvId) { + return withTx("rcv_queues", "readonly", async (tx) => { + const rq = await idbReq(tx.objectStore("rcv_queues").get([host, port, rcvId])) + if (!rq || !rq.deleted || toHex(rq.conn_id) !== toHex(connId)) return null + return rq + }) + }, + + // setConnectionNtfs — UPDATE connections SET enable_ntfs = ? WHERE conn_id = ? + async setConnectionNtfs(connId, enable) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn) { conn.enable_ntfs = enable ? 1 : 0; await idbReq(store.put(conn)) } + }) + }, + + // ============================================================ + // Subscriptions — AgentStore.hs:2211-2242, 943-954 + // ============================================================ + + // getSubscriptionServers (AgentStore.hs:2211-2226) + // SQL: SELECT DISTINCT c.user_id, q.host, q.port, COALESCE(q.server_key_hash, s.key_hash) + // FROM rcv_queues q + // JOIN servers s ON q.host = s.host AND q.port = s.port + // JOIN connections c ON q.conn_id = c.conn_id + // WHERE [q.to_subscribe = 1 AND] c.deleted = 0 AND q.deleted = 0 + async getSubscriptionServers(onlyNeeded) { + return withTx(["rcv_queues", "connections", "servers"], "readonly", async (tx) => { + const allQ = await idbReq(tx.objectStore("rcv_queues").getAll()) + const connStore = tx.objectStore("connections") + const srvStore = tx.objectStore("servers") + const seen = new Set() + const result: Array<{userId: number, host: string, port: string}> = [] + for (const q of allQ) { + if (q.deleted) continue + if (onlyNeeded && !q.to_subscribe) continue + const conn = await idbReq(connStore.get(q.conn_id)) + if (!conn || conn.deleted !== 0) continue + const keyHash = q.server_key_hash ?? (await idbReq(srvStore.get([q.host, q.port])))?.key_hash + const key = `${conn.user_id}:${q.host}:${q.port}:${keyHash ? toHex(keyHash) : ""}` + if (!seen.has(key)) { + seen.add(key) + result.push({userId: conn.user_id, host: q.host, port: q.port}) + } + } + return result + }) + }, + + // getUserServerRcvQueueSubs (AgentStore.hs:2228-2238) + // SQL: rcvQueueSubQuery WHERE [q.to_subscribe = 1 AND] c.deleted = 0 AND q.deleted = 0 + // AND c.user_id = ? AND q.host = ? AND q.port = ? AND COALESCE(q.server_key_hash, s.key_hash) = ? + // ORDER BY q.rcv_id LIMIT ? + async getUserServerRcvQueueSubs(userId, host, port, onlyNeeded, batchSize, cursor) { + return withTx(["rcv_queues", "connections", "servers"], "readonly", async (tx) => { + const allQ = await idbReq(tx.objectStore("rcv_queues").getAll()) + const connStore = tx.objectStore("connections") + const srvStore = tx.objectStore("servers") + // Filter matching queues + const matching: any[] = [] + for (const q of allQ) { + if (q.deleted) continue + if (q.host !== host || q.port !== port) continue + if (onlyNeeded && !q.to_subscribe) continue + if (cursor !== null && compareUint8Array(q.rcv_id, cursor as any) <= 0) continue + const conn = await idbReq(connStore.get(q.conn_id)) + if (!conn || conn.deleted !== 0 || conn.user_id !== userId) continue + matching.push(q) + } + // ORDER BY q.rcv_id + matching.sort((a, b) => compareUint8Array(a.rcv_id, b.rcv_id)) + // LIMIT + const queues = matching.slice(0, batchSize) + const nextCursor = queues.length === batchSize ? queues[queues.length - 1].rcv_id : null + return {queues, nextCursor} + }) + }, + + // unsetQueuesToSubscribe (AgentStore.hs:2240-2241) + // SQL: UPDATE rcv_queues SET to_subscribe = 0 WHERE to_subscribe = 1 + async unsetQueuesToSubscribe() { + return withTx("rcv_queues", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_queues") + const all = await idbReq(store.getAll()) + for (const q of all) { + if (q.to_subscribe === 1) { + q.to_subscribe = 0 + await idbReq(store.put(q)) + } + } + }) + }, + + // getConnectionsForDelivery (AgentStore.hs:943-945) + // SQL: SELECT DISTINCT conn_id FROM snd_message_deliveries WHERE failed = 0 + async getConnectionsForDelivery() { + return withTx("snd_message_deliveries", "readonly", async (tx) => { + const all = await idbReq(tx.objectStore("snd_message_deliveries").getAll()) + const seen = new Set() + const result: Uint8Array[] = [] + for (const d of all) { + if (d.failed) continue + const hex = toHex(d.conn_id) + if (!seen.has(hex)) { + seen.add(hex) + result.push(d.conn_id) + } + } + return result + }) + }, + + // getAllSndQueuesForDelivery (AgentStore.hs:947-954) + // SQL: sndQueueQuery + // JOIN (SELECT DISTINCT conn_id, snd_queue_id FROM snd_message_deliveries WHERE failed = 0) d + // ON d.conn_id = q.conn_id AND d.snd_queue_id = q.snd_queue_id + // WHERE c.deleted = 0 + async getAllSndQueuesForDelivery() { + return withTx(["snd_message_deliveries", "snd_queues", "connections"], "readonly", async (tx) => { + // Get distinct (conn_id, snd_queue_id) from non-failed deliveries + const allDel = await idbReq(tx.objectStore("snd_message_deliveries").getAll()) + const deliveryKeys = new Set() + for (const d of allDel) { + if (!d.failed) deliveryKeys.add(toHex(d.conn_id) + ":" + d.snd_queue_id) + } + // Get snd_queues that match + const allSQ = await idbReq(tx.objectStore("snd_queues").getAll()) + const connStore = tx.objectStore("connections") + const result: any[] = [] + for (const sq of allSQ) { + const key = toHex(sq.conn_id) + ":" + sq.snd_queue_id + if (!deliveryKeys.has(key)) continue + const conn = await idbReq(connStore.get(sq.conn_id)) + if (!conn || conn.deleted !== 0) continue + result.push(sq) + } + return result + }) + }, + + // ============================================================ + // Confirmations — AgentStore.hs:682-752 + // ============================================================ + + // createConfirmation (AgentStore.hs:682-691) + // SQL: INSERT INTO conn_confirmations + // (confirmation_id, conn_id, sender_key, e2e_snd_pub_key, ratchet_state, sender_conn_info, smp_reply_queues, smp_client_version, accepted) + // VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0) + async createConfirmation(confirmation) { + return withTx("conn_confirmations", "readwrite", async (tx) => { + await idbReq(tx.objectStore("conn_confirmations").add({ + confirmation_id: confirmation.confirmationId, + conn_id: confirmation.connId, + sender_key: confirmation.senderKey, + e2e_snd_pub_key: confirmation.e2eSndPubKey, + ratchet_state: confirmation.ratchetState, + sender_conn_info: confirmation.senderConnInfo, + smp_reply_queues: confirmation.smpReplyQueues, + smp_client_version: confirmation.smpClientVersion, + accepted: 0, + own_conn_info: null, + })) + return confirmation.confirmationId + }) + }, + + // acceptConfirmation (AgentStore.hs:693-721) + // SQL1: UPDATE conn_confirmations SET accepted = 1, own_conn_info = ? WHERE confirmation_id = ? + // SQL2: SELECT conn_id, ratchet_state, sender_key, e2e_snd_pub_key, sender_conn_info, smp_reply_queues, smp_client_version + // FROM conn_confirmations WHERE confirmation_id = ? + async acceptConfirmation(confirmationId, ownConnInfo) { + return withTx("conn_confirmations", "readwrite", async (tx) => { + const store = tx.objectStore("conn_confirmations") + const conf = await idbReq(store.get(confirmationId)) + if (!conf) throw new Error("confirmation not found") + conf.accepted = 1 + conf.own_conn_info = ownConnInfo + await idbReq(store.put(conf)) + return conf + }) + }, + + // getAcceptedConfirmation (AgentStore.hs:723-742) + // SQL: SELECT confirmation_id, ratchet_state, own_conn_info, sender_key, e2e_snd_pub_key, sender_conn_info, smp_reply_queues, smp_client_version + // FROM conn_confirmations WHERE conn_id = ? AND accepted = 1 + async getAcceptedConfirmation(connId) { + return withTx("conn_confirmations", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("conn_confirmations"), {conn_id: connId}) + const accepted = all.find((c: any) => c.accepted === 1) + return accepted ?? null + }) + }, + + // removeConfirmations (AgentStore.hs:744-752) + // SQL: DELETE FROM conn_confirmations WHERE conn_id = ? + async removeConfirmations(connId) { + return withTx("conn_confirmations", "readwrite", async (tx) => { + const store = tx.objectStore("conn_confirmations") + const allConf = await allByIndex(store, {conn_id: connId}) + const all = allConf.map((c: any) => c.confirmation_id) + for (const key of all) { + await idbReq(store.delete(key)) + } + }) + }, + + // ============================================================ + // Invitations — AgentStore.hs:754-799 + // ============================================================ + + // createInvitation (AgentStore.hs:754-763) + // SQL: INSERT INTO conn_invitations + // (invitation_id, contact_conn_id, cr_invitation, recipient_conn_info, accepted) + // VALUES (?, ?, ?, ?, 0) + async createInvitation(invitation) { + return withTx("conn_invitations", "readwrite", async (tx) => { + await idbReq(tx.objectStore("conn_invitations").add({ + invitation_id: invitation.invitationId, + contact_conn_id: invitation.contactConnId, + cr_invitation: invitation.crInvitation, + recipient_conn_info: invitation.recipientConnInfo, + accepted: 0, + own_conn_info: null, + })) + return invitation.invitationId + }) + }, + + // getInvitation (AgentStore.hs:765-779) + // SQL: SELECT contact_conn_id, cr_invitation, recipient_conn_info, own_conn_info, accepted + // FROM conn_invitations WHERE invitation_id = ? AND accepted = 0 + async getInvitation(invitationId) { + return withTx("conn_invitations", "readonly", async (tx) => { + const inv = await idbReq(tx.objectStore("conn_invitations").get(invitationId)) + if (!inv || inv.accepted !== 0) return null + return inv + }) + }, + + // acceptInvitation (AgentStore.hs:781-791) + // SQL: UPDATE conn_invitations SET accepted = 1, own_conn_info = ? WHERE invitation_id = ? + async acceptInvitation(invitationId, ownConnInfo) { + return withTx("conn_invitations", "readwrite", async (tx) => { + const store = tx.objectStore("conn_invitations") + const inv = await idbReq(store.get(invitationId)) + if (inv) { + inv.accepted = 1 + inv.own_conn_info = ownConnInfo + await idbReq(store.put(inv)) + } + }) + }, + + // unacceptInvitation (AgentStore.hs:793-795) + // SQL: UPDATE conn_invitations SET accepted = 0, own_conn_info = NULL WHERE invitation_id = ? + async unacceptInvitation(invitationId) { + return withTx("conn_invitations", "readwrite", async (tx) => { + const store = tx.objectStore("conn_invitations") + const inv = await idbReq(store.get(invitationId)) + if (inv) { + inv.accepted = 0 + inv.own_conn_info = null + await idbReq(store.put(inv)) + } + }) + }, + + // deleteInvitation (AgentStore.hs:797-799) + // SQL: DELETE FROM conn_invitations WHERE invitation_id = ? + async deleteInvitation(invitationId) { + return withTx("conn_invitations", "readwrite", async (tx) => { + await idbReq(tx.objectStore("conn_invitations").delete(invitationId)) + }) + }, + + // ============================================================ + // Messages — AgentStore.hs:873-1205 + // ============================================================ + + // updateRcvIds (AgentStore.hs:873-879) + // Calls retrieveLastIdsAndHashRcv_ (AgentStore.hs:2584-2599): + // SQL: SELECT last_internal_msg_id, last_internal_rcv_msg_id, last_external_snd_msg_id, last_rcv_msg_hash FROM connections WHERE conn_id = ? + // Then updateLastIdsRcv_ (AgentStore.hs:2601-2611): + // SQL: UPDATE connections SET last_internal_msg_id = ?, last_internal_rcv_msg_id = ? WHERE conn_id = ? + async updateRcvIds(connId) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (!conn) throw new Error("connection not found") + const internalId = conn.last_internal_msg_id + 1 + const internalRcvId = conn.last_internal_rcv_msg_id + 1 + const prevExternalSndId = conn.last_external_snd_msg_id + const prevRcvMsgHash = conn.last_rcv_msg_hash + conn.last_internal_msg_id = internalId + conn.last_internal_rcv_msg_id = internalRcvId + await idbReq(store.put(conn)) + return {internalId, internalRcvId, prevExternalSndId, prevRcvMsgHash} + }) + }, + + // createRcvMsg (AgentStore.hs:881-886) + // Calls insertRcvMsgBase_ (AgentStore.hs:2615-2625): + // SQL: INSERT INTO messages (conn_id, internal_id, internal_ts, internal_rcv_id, internal_snd_id, msg_type, msg_flags, msg_body, pq_encryption) VALUES (?,?,?,?,?,?,?,?,?) + // Calls insertRcvMsgDetails_ (AgentStore.hs:2627-2641): + // SQL: INSERT INTO rcv_messages (conn_id, rcv_queue_id, internal_rcv_id, internal_id, external_snd_id, broker_id, broker_ts, internal_hash, external_prev_snd_hash, integrity) VALUES (?,?,?,?,?,?,?,?,?,?) + // SQL: INSERT INTO encrypted_rcv_message_hashes (conn_id, hash) VALUES (?,?) + // Calls updateRcvMsgHash (AgentStore.hs:2643-2655): + // SQL: UPDATE connections SET last_external_snd_msg_id = ?, last_rcv_msg_hash = ? WHERE conn_id = ? AND last_internal_rcv_msg_id = ? + // Calls setLastBrokerTs (AgentStore.hs:888-890): + // SQL: UPDATE rcv_queues SET last_broker_ts = ? WHERE conn_id = ? AND rcv_queue_id = ? AND (last_broker_ts IS NULL OR last_broker_ts < ?) + async createRcvMsg(connId, rcvQueue, rcvMsgData) { + return withTx(["messages", "rcv_messages", "encrypted_rcv_message_hashes", "connections", "rcv_queues"], "readwrite", async (tx) => { + const {msgMeta, msgType, msgFlags, msgBody, internalRcvId, internalHash, externalPrevSndHash, encryptedMsgHash} = rcvMsgData + const {recipient, broker, sndMsgId, integrity, pqEncryption} = msgMeta + const [internalId, internalTs] = recipient + const [brokerId, brokerTs] = broker + + // insertRcvMsgBase_: INSERT INTO messages + await idbReq(tx.objectStore("messages").add({ + conn_id: connId, internal_id: internalId, internal_ts: internalTs, + internal_rcv_id: internalRcvId, internal_snd_id: null, + msg_type: msgType, msg_flags: msgFlags, msg_body: msgBody, pq_encryption: pqEncryption, + })) + + // insertRcvMsgDetails_: INSERT INTO rcv_messages + await idbReq(tx.objectStore("rcv_messages").add({ + conn_id: connId, rcv_queue_id: rcvQueue.dbQueueId, internal_rcv_id: internalRcvId, + internal_id: internalId, external_snd_id: sndMsgId, + broker_id: brokerId, broker_ts: brokerTs, + internal_hash: internalHash, external_prev_snd_hash: externalPrevSndHash, + integrity, user_ack: 0, receive_attempts: 0, + })) + + // insertRcvMsgDetails_: INSERT INTO encrypted_rcv_message_hashes + await idbReq(tx.objectStore("encrypted_rcv_message_hashes").add({ + conn_id: connId, hash: encryptedMsgHash, created_at: new Date().toISOString(), + })) + + // updateRcvMsgHash: UPDATE connections SET last_external_snd_msg_id, last_rcv_msg_hash + const connStore = tx.objectStore("connections") + const conn = await idbReq(connStore.get(connId)) + if (conn && conn.last_internal_rcv_msg_id === internalRcvId) { + conn.last_external_snd_msg_id = sndMsgId + conn.last_rcv_msg_hash = internalHash + await idbReq(connStore.put(conn)) + } + + // setLastBrokerTs: UPDATE rcv_queues SET last_broker_ts + const rcvQStore = tx.objectStore("rcv_queues") + const allRQ = await allByIndex(rcvQStore, {conn_id: connId}) + for (const rq of allRQ) { + if (rq.rcv_queue_id === rcvQueue.dbQueueId) { + if (rq.last_broker_ts == null || rq.last_broker_ts < brokerTs) { + rq.last_broker_ts = brokerTs + await idbReq(rcvQStore.put(rq)) + } + break + } + } + }) + }, + + // setLastBrokerTs (AgentStore.hs:888-890) + // SQL: UPDATE rcv_queues SET last_broker_ts = ? WHERE conn_id = ? AND rcv_queue_id = ? AND (last_broker_ts IS NULL OR last_broker_ts < ?) + async setLastBrokerTs(connId, dbQueueId, brokerTs) { + return withTx("rcv_queues", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_queues") + const all = await allByIndex(store, {conn_id: connId}) + for (const rq of all) { + if (rq.rcv_queue_id === dbQueueId) { + if (rq.last_broker_ts == null || rq.last_broker_ts < brokerTs) { + rq.last_broker_ts = brokerTs + await idbReq(store.put(rq)) + } + break + } + } + }) + }, + + // updateRcvMsgHash (AgentStore.hs:2643-2655) + // SQL: UPDATE connections SET last_external_snd_msg_id = ?, last_rcv_msg_hash = ? + // WHERE conn_id = ? AND last_internal_rcv_msg_id = ? + async updateRcvMsgHash(connId, sndMsgId, internalRcvId, hash) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn && conn.last_internal_rcv_msg_id === internalRcvId) { + conn.last_external_snd_msg_id = sndMsgId + conn.last_rcv_msg_hash = hash + await idbReq(store.put(conn)) + } + }) + }, + + // createSndMsgBody (AgentStore.hs:892-898) + // SQL: INSERT INTO snd_message_bodies (agent_msg) VALUES (?) RETURNING snd_message_body_id + async createSndMsgBody(agentMsg) { + return withTx("snd_message_bodies", "readwrite", async (tx) => { + return await idbReq(tx.objectStore("snd_message_bodies").add({agent_msg: agentMsg})) as number + }) + }, + + // updateSndIds (AgentStore.hs:900-906) + // Calls retrieveLastIdsAndHashSnd_ (AgentStore.hs:2659-2673): + // SQL: SELECT last_internal_msg_id, last_internal_snd_msg_id, last_snd_msg_hash FROM connections WHERE conn_id = ? + // Then updateLastIdsSnd_ (AgentStore.hs:2675-2685): + // SQL: UPDATE connections SET last_internal_msg_id = ?, last_internal_snd_msg_id = ? WHERE conn_id = ? + async updateSndIds(connId) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (!conn) throw new Error("connection not found") + const internalId = conn.last_internal_msg_id + 1 + const internalSndId = conn.last_internal_snd_msg_id + 1 + const prevSndMsgHash = conn.last_snd_msg_hash + conn.last_internal_msg_id = internalId + conn.last_internal_snd_msg_id = internalSndId + await idbReq(store.put(conn)) + return {internalId, internalSndId, prevSndMsgHash} + }) + }, + + // createSndMsg (AgentStore.hs:908-912) + // Calls insertSndMsgBase_ (AgentStore.hs:2689-2699): + // SQL: INSERT INTO messages (conn_id, internal_id, internal_ts, internal_rcv_id, internal_snd_id, msg_type, msg_flags, msg_body, pq_encryption) VALUES (?,?,?,?,?,?,?,?,?) + // Calls insertSndMsgDetails_ (AgentStore.hs:2701-2715): + // SQL: INSERT INTO snd_messages (conn_id, internal_snd_id, internal_id, internal_hash, previous_msg_hash, msg_encrypt_key, padded_msg_len, snd_message_body_id) VALUES (?,?,?,?,?,?,?,?) + // Calls updateSndMsgHash (AgentStore.hs:2717-2728): + // SQL: UPDATE connections SET last_snd_msg_hash = ? WHERE conn_id = ? AND last_internal_snd_msg_id = ? + async createSndMsg(connId, sndMsgData) { + return withTx(["messages", "snd_messages", "connections"], "readwrite", async (tx) => { + const {internalId, internalSndId, internalTs, msgType, msgFlags, msgBody, pqEncryption, + internalHash, prevMsgHash, msgEncryptKey, paddedMsgLen, sndMessageBodyId} = sndMsgData + + // insertSndMsgBase_: INSERT INTO messages + await idbReq(tx.objectStore("messages").add({ + conn_id: connId, internal_id: internalId, internal_ts: internalTs, + internal_rcv_id: null, internal_snd_id: internalSndId, + msg_type: msgType, msg_flags: msgFlags, msg_body: msgBody, pq_encryption: pqEncryption, + })) + + // insertSndMsgDetails_: INSERT INTO snd_messages + await idbReq(tx.objectStore("snd_messages").add({ + conn_id: connId, internal_snd_id: internalSndId, internal_id: internalId, + internal_hash: internalHash, previous_msg_hash: prevMsgHash, + msg_encrypt_key: msgEncryptKey, padded_msg_len: paddedMsgLen, + snd_message_body_id: sndMessageBodyId, + retry_int_slow: null, retry_int_fast: null, + rcpt_internal_id: null, rcpt_status: null, + })) + + // updateSndMsgHash: UPDATE connections SET last_snd_msg_hash + const connStore = tx.objectStore("connections") + const conn = await idbReq(connStore.get(connId)) + if (conn && conn.last_internal_snd_msg_id === internalSndId) { + conn.last_snd_msg_hash = internalHash + await idbReq(connStore.put(conn)) + } + }) + }, + + // updateSndMsgHash (AgentStore.hs:2717-2728) + // SQL: UPDATE connections SET last_snd_msg_hash = ? WHERE conn_id = ? AND last_internal_snd_msg_id = ? + async updateSndMsgHash(connId, internalSndId, hash) { + return withTx("connections", "readwrite", async (tx) => { + const store = tx.objectStore("connections") + const conn = await idbReq(store.get(connId)) + if (conn && conn.last_internal_snd_msg_id === internalSndId) { + conn.last_snd_msg_hash = hash + await idbReq(store.put(conn)) + } + }) + }, + + // createSndMsgDelivery (AgentStore.hs:914-916) + // SQL: INSERT INTO snd_message_deliveries (conn_id, snd_queue_id, internal_id) VALUES (?, ?, ?) + async createSndMsgDelivery(connId, sndQueue, internalId) { + return withTx("snd_message_deliveries", "readwrite", async (tx) => { + await idbReq(tx.objectStore("snd_message_deliveries").add({ + conn_id: connId, snd_queue_id: sndQueue.dbQueueId, internal_id: internalId, failed: 0, + })) + }) + }, + + // getPendingQueueMsg (AgentStore.hs:956-1002) + // getMsgId SQL: SELECT internal_id FROM snd_message_deliveries d + // WHERE conn_id = ? AND snd_queue_id = ? AND failed = 0 + // ORDER BY internal_id ASC LIMIT 1 + // getMsgData SQL: SELECT m.msg_type, m.msg_flags, m.msg_body, m.pq_encryption, m.internal_ts, m.internal_snd_id, s.previous_msg_hash, + // s.retry_int_slow, s.retry_int_fast, s.msg_encrypt_key, s.padded_msg_len, sb.agent_msg + // FROM messages m + // JOIN snd_messages s ON s.conn_id = m.conn_id AND s.internal_id = m.internal_id + // LEFT JOIN snd_message_bodies sb ON sb.snd_message_body_id = s.snd_message_body_id + // WHERE m.conn_id = ? AND m.internal_id = ? + async getPendingQueueMsg(connId, sndQueue) { + return withTx(["snd_message_deliveries", "messages", "snd_messages", "snd_message_bodies"], "readonly", async (tx) => { + // getMsgId: find first non-failed delivery for this queue + const allDel = await allByIndex(tx.objectStore("snd_message_deliveries"), {conn_id: connId, snd_queue_id: sndQueue.dbQueueId}) + const pending = allDel.filter((d: any) => !d.failed).sort((a: any, b: any) => a.internal_id - b.internal_id) + if (pending.length === 0) return null + const msgId = pending[0].internal_id + + // getMsgData: JOIN messages + snd_messages + snd_message_bodies + const msg = await idbReq(tx.objectStore("messages").get([connId, msgId])) + if (!msg) return null + // Find snd_message by [conn_id, internal_id] via index + const sndMsgs = await allByIndex(tx.objectStore("snd_messages"), {conn_id: connId, internal_id: msgId}) + if (sndMsgs.length === 0) return null + const sm = sndMsgs[0] + // LEFT JOIN snd_message_bodies + let sndMsgBody: Uint8Array | null = null + if (sm.snd_message_body_id != null) { + const body = await idbReq(tx.objectStore("snd_message_bodies").get(sm.snd_message_body_id)) + if (body) sndMsgBody = body.agent_msg + } + + return { + connId, sndQueueId: sndQueue.dbQueueId, internalId: msgId, + msgType: msg.msg_type, msgFlags: msg.msg_flags, msgBody: msg.msg_body, + internalHash: sm.internal_hash, prevMsgHash: sm.previous_msg_hash, + pqEncryption: msg.pq_encryption, + msgEncryptKey: sm.msg_encrypt_key, paddedMsgLen: sm.padded_msg_len, + sndMessageBodyId: sm.snd_message_body_id, + } + }) + }, + + // updatePendingMsgRIState (AgentStore.hs:1024-1026) + // SQL: UPDATE snd_messages SET retry_int_slow = ?, retry_int_fast = ? WHERE conn_id = ? AND internal_id = ? + async updatePendingMsgRIState(connId, msgId, retryIntSlow, retryIntFast) { + return withTx("snd_messages", "readwrite", async (tx) => { + const store = tx.objectStore("snd_messages") + const all = await allByIndex(store, {conn_id: connId, internal_id: msgId}) + if (all.length > 0) { + const sm = all[0] + sm.retry_int_slow = retryIntSlow + sm.retry_int_fast = retryIntFast + await idbReq(store.put(sm)) + } + }) + }, + + // setMsgUserAck (AgentStore.hs:1059-1073) + // SQL1: SELECT rcv_queue_id, broker_id FROM rcv_messages WHERE conn_id = ? AND internal_id = ? + // SQL2: UPDATE rcv_messages SET user_ack = 1 WHERE conn_id = ? AND internal_id = ? + // Then getRcvQueueById to return the rcv queue + async setMsgUserAck(connId, internalId) { + return withTx(["rcv_messages", "rcv_queues"], "readwrite", async (tx) => { + const rmStore = tx.objectStore("rcv_messages") + // Find rcv_message by conn_id + internal_id via index + const all = await allByIndex(rmStore, {conn_id: connId, internal_id: internalId}) + if (all.length === 0) throw new Error("rcv message not found") + const rm = all[0] + const dbRcvId = rm.rcv_queue_id + const brokerId = rm.broker_id + // Update user_ack + rm.user_ack = 1 + await idbReq(rmStore.put(rm)) + // getRcvQueueById: find rcv_queue by conn_id + rcv_queue_id + const rqStore = tx.objectStore("rcv_queues") + const rqs = await allByIndex(rqStore, {conn_id: connId, rcv_queue_id: dbRcvId}) + if (rqs.length === 0) throw new Error("rcv queue not found") + return {rcvQueue: rqs[0], brokerId} + }) + }, + + // getRcvMsg (AgentStore.hs:1075-1089) + // SQL: SELECT r.internal_id, m.internal_ts, r.broker_id, r.broker_ts, r.external_snd_id, r.integrity, r.internal_hash, + // m.msg_type, m.msg_body, m.pq_encryption, s.internal_id, s.rcpt_status, r.user_ack + // FROM rcv_messages r + // JOIN messages m ON r.conn_id = m.conn_id AND r.internal_id = m.internal_id + // LEFT JOIN snd_messages s ON s.conn_id = r.conn_id AND s.rcpt_internal_id = r.internal_id + // WHERE r.conn_id = ? AND r.internal_id = ? + async getRcvMsg(connId, internalId) { + return withTx(["rcv_messages", "messages", "snd_messages"], "readonly", async (tx) => { + // Find rcv_message + const rmAll = await allByIndex(tx.objectStore("rcv_messages"), {conn_id: connId, internal_id: internalId}) + if (rmAll.length === 0) return null + const rm = rmAll[0] + // JOIN messages + const msg = await idbReq(tx.objectStore("messages").get([connId, internalId])) + if (!msg) return null + // LEFT JOIN snd_messages ON rcpt_internal_id = r.internal_id + const sndAll = await allByIndex(tx.objectStore("snd_messages"), {conn_id: connId, internal_id: internalId}) + const sndRcpt = sndAll.find((s: any) => s.rcpt_internal_id === internalId) + return { + internalId: rm.internal_id, + msgMeta: { + integrity: rm.integrity, + recipient: [rm.internal_id, msg.internal_ts], + broker: [rm.broker_id, rm.broker_ts], + sndMsgId: rm.external_snd_id, + pqEncryption: msg.pq_encryption, + }, + msgType: msg.msg_type, + msgBody: msg.msg_body, + userAck: rm.user_ack === 1, + msgReceipt: sndRcpt?.rcpt_status ?? null, + } + }) + }, + + // getLastMsg (AgentStore.hs:1091-1106) + // SQL: SELECT ... FROM rcv_messages r + // JOIN messages m ON r.conn_id = m.conn_id AND r.internal_id = m.internal_id + // JOIN connections c ON r.conn_id = c.conn_id AND c.last_internal_msg_id = r.internal_id + // LEFT JOIN snd_messages s ON s.conn_id = r.conn_id AND s.rcpt_internal_id = r.internal_id + // WHERE r.conn_id = ? AND r.broker_id = ? + async getLastMsg(connId, brokerId) { + return withTx(["rcv_messages", "messages", "connections", "snd_messages"], "readonly", async (tx) => { + // Find rcv_message by broker_id + const rmAll = await allByIndex(tx.objectStore("rcv_messages"), {conn_id: connId, broker_id: brokerId}) + if (rmAll.length === 0) return null + const rm = rmAll[0] + // JOIN connections: verify last_internal_msg_id = r.internal_id + const conn = await idbReq(tx.objectStore("connections").get(connId)) + if (!conn || conn.last_internal_msg_id !== rm.internal_id) return null + // JOIN messages + const msg = await idbReq(tx.objectStore("messages").get([connId, rm.internal_id])) + if (!msg) return null + return { + internalId: rm.internal_id, + msgMeta: { + integrity: rm.integrity, + recipient: [rm.internal_id, msg.internal_ts], + broker: [rm.broker_id, rm.broker_ts], + sndMsgId: rm.external_snd_id, + pqEncryption: msg.pq_encryption, + }, + msgType: msg.msg_type, + msgBody: msg.msg_body, + userAck: rm.user_ack === 1, + msgReceipt: null, + } + }) + }, + + // incMsgRcvAttempts (AgentStore.hs:1114-1125) + // SQL: UPDATE rcv_messages SET receive_attempts = receive_attempts + 1 + // WHERE conn_id = ? AND internal_id = ? RETURNING receive_attempts + async incMsgRcvAttempts(connId, internalId) { + return withTx("rcv_messages", "readwrite", async (tx) => { + const store = tx.objectStore("rcv_messages") + const all = await allByIndex(store, {conn_id: connId, internal_id: internalId}) + if (all.length === 0) throw new Error("rcv message not found") + const rm = all[0] + rm.receive_attempts = (rm.receive_attempts || 0) + 1 + await idbReq(store.put(rm)) + return rm.receive_attempts + }) + }, + + // checkRcvMsgHashExists (AgentStore.hs:1127-1133) + // SQL: SELECT 1 FROM encrypted_rcv_message_hashes WHERE conn_id = ? AND hash = ? LIMIT 1 + async checkRcvMsgHashExists(connId, hash) { + return withTx("encrypted_rcv_message_hashes", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("encrypted_rcv_message_hashes"), {conn_id: connId, hash}) + return all.length > 0 + }) + }, + + // getRcvMsgBrokerTs (AgentStore.hs:1135-1138) + // SQL: SELECT broker_ts FROM rcv_messages WHERE conn_id = ? AND broker_id = ? + async getRcvMsgBrokerTs(connId, brokerId) { + return withTx("rcv_messages", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("rcv_messages"), {conn_id: connId, broker_id: brokerId}) + if (all.length === 0) return null + return all[0].broker_ts + }) + }, + + // deleteMsg (AgentStore.hs:1140-1142) + // SQL: DELETE FROM messages WHERE conn_id = ? AND internal_id = ? + async deleteMsg(connId, internalId) { + return withTx("messages", "readwrite", async (tx) => { + await idbReq(tx.objectStore("messages").delete([connId, internalId])) + }) + }, + + // deleteDeliveredSndMsg (AgentStore.hs:1153-1159) + // SQL: countPendingSndDeliveries_ then deleteMsg if cnt == 0 + // countPendingSndDeliveries_: SELECT count(*) FROM snd_message_deliveries WHERE conn_id = ? AND internal_id = ? AND failed = 0 + async deleteDeliveredSndMsg(connId, internalId) { + return withTx(["snd_message_deliveries", "messages"], "readwrite", async (tx) => { + const allDel = await idbReq(tx.objectStore("snd_message_deliveries").getAll()) + const cnt = allDel.filter((d: any) => toHex(d.conn_id) === toHex(connId) && d.internal_id === internalId && !d.failed).length + if (cnt === 0) { + await idbReq(tx.objectStore("messages").delete([connId, internalId])) + } + }) + }, + + // deleteSndMsgDelivery (AgentStore.hs:1161-1205) + // SQL1: DELETE FROM snd_message_deliveries WHERE conn_id = ? AND snd_queue_id = ? AND internal_id = ? + // SQL2: SELECT rcpt_status, snd_message_body_id FROM snd_messages + // WHERE NOT EXISTS (SELECT 1 FROM snd_message_deliveries WHERE conn_id = ? AND internal_id = ? AND failed = 0) + // AND conn_id = ? AND internal_id = ? + // Then conditionally: DELETE FROM messages, DELETE FROM snd_message_bodies + async deleteSndMsgDelivery(connId, sndQueue, msgId, keepForReceipt) { + return withTx(["snd_message_deliveries", "snd_messages", "messages", "snd_message_bodies"], "readwrite", async (tx) => { + // Delete the delivery record + const delStore = tx.objectStore("snd_message_deliveries") + const allDel = await idbReq(delStore.getAll()) + for (const d of allDel) { + if (toHex(d.conn_id) === toHex(connId) && d.snd_queue_id === sndQueue.dbQueueId && d.internal_id === msgId) { + await idbReq(delStore.delete(d.snd_message_delivery_id)) + break + } + } + + // Check if there are remaining non-failed deliveries + const remainingDel = await idbReq(delStore.getAll()) + const hasPending = remainingDel.some((d: any) => toHex(d.conn_id) === toHex(connId) && d.internal_id === msgId && !d.failed) + if (hasPending) return + + // Get snd_message for this msg + const smAll = await allByIndex(tx.objectStore("snd_messages"), {conn_id: connId, internal_id: msgId}) + if (smAll.length === 0) return + const sm = smAll[0] + const rcptStatus = sm.rcpt_status + const sndMsgBodyId = sm.snd_message_body_id + + // Decide whether to delete or clear content + // MROk → deleteMsg; otherwise if keepForReceipt → deleteMsgContent; else deleteMsg + const shouldDeleteFull = rcptStatus === "ok" || !keepForReceipt + if (shouldDeleteFull) { + await idbReq(tx.objectStore("messages").delete([connId, msgId])) + } else { + // deleteMsgContent: clear msg_body + const msg = await idbReq(tx.objectStore("messages").get([connId, msgId])) + if (msg) { + msg.msg_body = new Uint8Array(0) + await idbReq(tx.objectStore("messages").put(msg)) + } + sm.snd_message_body_id = null + await idbReq(tx.objectStore("snd_messages").put(sm)) + } + + // Delete snd_message_body if no other snd_messages reference it + if (sndMsgBodyId != null) { + const allSM = await idbReq(tx.objectStore("snd_messages").getAll()) + const referenced = allSM.some((s: any) => s.snd_message_body_id === sndMsgBodyId) + if (!referenced) { + await idbReq(tx.objectStore("snd_message_bodies").delete(sndMsgBodyId)) + } + } + }) + }, + + // getSndMsgViaRcpt (AgentStore.hs:918-934) + // SQL: SELECT s.internal_id, m.msg_type, s.internal_hash, s.rcpt_internal_id, s.rcpt_status + // FROM snd_messages s + // JOIN messages m ON s.conn_id = m.conn_id AND s.internal_id = m.internal_id + // WHERE s.conn_id = ? AND s.internal_snd_id = ? + async getSndMsgViaRcpt(connId, sndMsgId) { + return withTx(["snd_messages", "messages"], "readonly", async (tx) => { + const sm = await idbReq(tx.objectStore("snd_messages").get([connId, sndMsgId])) + if (!sm) return null + const msg = await idbReq(tx.objectStore("messages").get([connId, sm.internal_id])) + if (!msg) return null + return { + internalId: sm.internal_id, + msgType: msg.msg_type, + internalHash: sm.internal_hash, + msgReceipt: sm.rcpt_status ?? null, + } + }) + }, + + // updateSndMsgRcpt (AgentStore.hs:936-941) + // SQL: UPDATE snd_messages SET rcpt_internal_id = ?, rcpt_status = ? WHERE conn_id = ? AND internal_snd_id = ? + async updateSndMsgRcpt(connId, sndMsgId, receipt) { + return withTx("snd_messages", "readwrite", async (tx) => { + const store = tx.objectStore("snd_messages") + const sm = await idbReq(store.get([connId, sndMsgId])) + if (sm) { + sm.rcpt_internal_id = receipt + sm.rcpt_status = receipt + await idbReq(store.put(sm)) + } + }) + }, + + // ============================================================ + // Ratchet — AgentStore.hs:1246-1367 + // ============================================================ + + // createRatchetX3dhKeys (AgentStore.hs:1246-1248) + // SQL: INSERT INTO ratchets (conn_id, x3dh_priv_key_1, x3dh_priv_key_2, pq_priv_kem) VALUES (?, ?, ?, ?) + async createRatchetX3dhKeys(connId, privKey1, privKey2, pqKem) { + return withTx("ratchets", "readwrite", async (tx) => { + await idbReq(tx.objectStore("ratchets").add({ + conn_id: connId, + x3dh_priv_key_1: privKey1, x3dh_priv_key_2: privKey2, + pq_priv_kem: pqKem, + ratchet_state: null, + x3dh_pub_key_1: null, x3dh_pub_key_2: null, pq_pub_kem: null, + })) + }) + }, + + // getRatchetX3dhKeys (AgentStore.hs:1250-1257) + // SQL: SELECT x3dh_priv_key_1, x3dh_priv_key_2, pq_priv_kem FROM ratchets WHERE conn_id = ? + async getRatchetX3dhKeys(connId) { + return withTx("ratchets", "readonly", async (tx) => { + const r = await idbReq(tx.objectStore("ratchets").get(connId)) + if (!r || !r.x3dh_priv_key_1 || !r.x3dh_priv_key_2) return null + return {privKey1: r.x3dh_priv_key_1, privKey2: r.x3dh_priv_key_2, pqKem: r.pq_priv_kem} + }) + }, + + // createRatchet (AgentStore.hs:1303-1319) + // SQL: INSERT INTO ratchets (conn_id, ratchet_state) VALUES (?, ?) + // ON CONFLICT (conn_id) DO UPDATE SET ratchet_state = ?, + // x3dh_priv_key_1 = NULL, x3dh_priv_key_2 = NULL, x3dh_pub_key_1 = NULL, x3dh_pub_key_2 = NULL, + // pq_priv_kem = NULL, pq_pub_kem = NULL + async createRatchet(connId, ratchetState) { + return withTx("ratchets", "readwrite", async (tx) => { + // ON CONFLICT DO UPDATE → use put (upsert) + await idbReq(tx.objectStore("ratchets").put({ + conn_id: connId, + ratchet_state: ratchetState, + x3dh_priv_key_1: null, x3dh_priv_key_2: null, + x3dh_pub_key_1: null, x3dh_pub_key_2: null, + pq_priv_kem: null, pq_pub_kem: null, + })) + }) + }, + + // getRatchet (AgentStore.hs:1334-1345) + // SQL: SELECT ratchet_state FROM ratchets WHERE conn_id = ? + async getRatchet(connId) { + return withTx("ratchets", "readonly", async (tx) => { + const r = await idbReq(tx.objectStore("ratchets").get(connId)) + return r?.ratchet_state ?? null + }) + }, + + // getRatchetForUpdate (AgentStore.hs:1325-1332) — same as getRatchet in IndexedDB (single-threaded) + async getRatchetForUpdate(connId) { + return this.getRatchet(connId) + }, + + // getSkippedMsgKeys (AgentStore.hs:1347-1354) + // SQL: SELECT header_key, msg_n, msg_key FROM skipped_messages WHERE conn_id = ? + // Returns: Map> + async getSkippedMsgKeys(connId) { + return withTx("skipped_messages", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("skipped_messages"), {conn_id: connId}) + const result = new Map>() + for (const row of all) { + const hkHex = toHex(row.header_key) + let inner = result.get(hkHex) + if (!inner) { + inner = new Map() + result.set(hkHex, inner) + } + inner.set(row.msg_n, row.msg_key) + } + return result + }) + }, + + // updateRatchet (AgentStore.hs:1356-1366) + // SQL1: UPDATE ratchets SET ratchet_state = ? WHERE conn_id = ? + // SMDNoChange: no-op + // SMDRemove: DELETE FROM skipped_messages WHERE conn_id = ? AND header_key = ? AND msg_n = ? + // SMDAdd: INSERT INTO skipped_messages (conn_id, header_key, msg_n, msg_key) VALUES (?, ?, ?, ?) + async updateRatchet(connId, ratchetState, skippedMsgDiff) { + return withTx(["ratchets", "skipped_messages"], "readwrite", async (tx) => { + // Update ratchet state + const rStore = tx.objectStore("ratchets") + const r = await idbReq(rStore.get(connId)) + if (r) { + r.ratchet_state = ratchetState + await idbReq(rStore.put(r)) + } + // Apply skipped message diff + const diff = skippedMsgDiff as any + if (!diff || diff.type === "noChange") return + const skStore = tx.objectStore("skipped_messages") + if (diff.type === "remove") { + // Delete specific skipped message + const all = await allByIndex(skStore, {conn_id: connId}) + for (const row of all) { + if (toHex(row.header_key) === toHex(diff.headerKey) && row.msg_n === diff.msgN) { + await idbReq(skStore.delete(row.skipped_message_id)) + break + } + } + } else if (diff.type === "add") { + // Add new skipped message keys + for (const [hk, mks] of diff.keys.entries()) { + for (const [msgN, mk] of mks.entries()) { + await idbReq(skStore.add({ + conn_id: connId, header_key: hk, msg_n: msgN, msg_key: mk, + })) + } + } + } + }) + }, + + // ============================================================ + // Commands — AgentStore.hs:1368-1497 + // ============================================================ + + // createCommand (AgentStore.hs:1368-1392) + // SQL: INSERT INTO commands (host, port, corr_id, conn_id, command_tag, command, server_key_hash, created_at) VALUES (?,?,?,?,?,?,?,?) + async createCommand(corrId, connId, host, port, command) { + return withTx("commands", "readwrite", async (tx) => { + const id = await idbReq(tx.objectStore("commands").add({ + host, port, corr_id: corrId, conn_id: connId, + command_tag: command.commandTag, command: command.command, + agent_version: command.agentVersion, + server_key_hash: command.serverKeyHash, + created_at: new Date().toISOString(), failed: 0, + })) + return id as number + }) + }, + + // getPendingCommandServers (AgentStore.hs:1403-1422) + // SQL: SELECT DISTINCT c.conn_id, c.host, c.port, COALESCE(c.server_key_hash, s.key_hash) + // FROM commands c LEFT JOIN servers s ON s.host = c.host AND s.port = c.port + // ORDER BY c.conn_id + // Then filter by connIds set membership + async getPendingCommandServers(connIds) { + return withTx(["commands", "servers"], "readonly", async (tx) => { + const allCmds = await idbReq(tx.objectStore("commands").getAll()) + const connIdSet = new Set(connIds.map(toHex)) + const seen = new Set() + const result: Array<{connId: Uint8Array, host: string, port: string}> = [] + for (const cmd of allCmds) { + if (!cmd.host || !cmd.port) continue + if (!connIdSet.has(toHex(cmd.conn_id))) continue + const key = toHex(cmd.conn_id) + ":" + cmd.host + ":" + cmd.port + if (!seen.has(key)) { + seen.add(key) + result.push({connId: cmd.conn_id, host: cmd.host, port: cmd.port}) + } + } + return result + }) + }, + + // getAllPendingCommandConns (AgentStore.hs:1424-1437) + // SQL: SELECT DISTINCT c.conn_id, c.host, c.port, COALESCE(c.server_key_hash, s.key_hash) + // FROM commands c + // JOIN connections cs ON c.conn_id = cs.conn_id + // LEFT JOIN servers s ON s.host = c.host AND s.port = c.port + // WHERE cs.deleted = 0 + async getAllPendingCommandConns() { + return withTx(["commands", "connections", "servers"], "readonly", async (tx) => { + const allCmds = await idbReq(tx.objectStore("commands").getAll()) + const connStore = tx.objectStore("connections") + const seen = new Set() + const result: Array<{connId: Uint8Array, host: string, port: string}> = [] + for (const cmd of allCmds) { + if (!cmd.host || !cmd.port) continue + const conn = await idbReq(connStore.get(cmd.conn_id)) + if (!conn || conn.deleted !== 0) continue + const key = toHex(cmd.conn_id) + ":" + cmd.host + ":" + cmd.port + if (!seen.has(key)) { + seen.add(key) + result.push({connId: cmd.conn_id, host: cmd.host, port: cmd.port}) + } + } + return result + }) + }, + + // getPendingServerCommand (AgentStore.hs:1439-1480) + // getCmdId SQL: SELECT command_id FROM commands + // WHERE conn_id = ? AND host = ? AND port = ? AND failed = 0 + // ORDER BY created_at ASC, command_id ASC LIMIT 1 + // getCommand SQL: SELECT c.corr_id, cs.user_id, c.command FROM commands c + // JOIN connections cs USING (conn_id) WHERE c.command_id = ? + async getPendingServerCommand(host, port) { + return withTx(["commands", "connections"], "readonly", async (tx) => { + const allCmds = await allByIndex(tx.objectStore("commands"), {host, port}) + // Filter non-failed, sort by created_at then command_id + const pending = allCmds + .filter((c: any) => !c.failed) + .sort((a: any, b: any) => { + const tsCompare = (a.created_at || "").localeCompare(b.created_at || "") + return tsCompare !== 0 ? tsCompare : (a.command_id - b.command_id) + }) + if (pending.length === 0) return null + return pending[0] + }) + }, + + // updateCommandServer (AgentStore.hs:1482-1493) + // SQL: UPDATE commands SET host = ?, port = ?, server_key_hash = ? WHERE command_id = ? + async updateCommandServer(commandId, host, port) { + return withTx("commands", "readwrite", async (tx) => { + const store = tx.objectStore("commands") + const cmd = await idbReq(store.get(commandId)) + if (cmd) { + cmd.host = host + cmd.port = port + await idbReq(store.put(cmd)) + } + }) + }, + + // deleteCommand (AgentStore.hs:1495-1497) + // SQL: DELETE FROM commands WHERE command_id = ? + async deleteCommand(commandId) { + return withTx("commands", "readwrite", async (tx) => { + await idbReq(tx.objectStore("commands").delete(commandId)) + }) + }, + + // ============================================================ + // Encrypted message hash dedup — AgentStore.hs:1127-1133, 2641 + // ============================================================ + + // checkRcvMsgHashExists_encrypted (AgentStore.hs:1127-1133) + // SQL: SELECT 1 FROM encrypted_rcv_message_hashes WHERE conn_id = ? AND hash = ? LIMIT 1 + async checkRcvMsgHashExists_encrypted(connId, hash) { + return withTx("encrypted_rcv_message_hashes", "readonly", async (tx) => { + const all = await allByIndex(tx.objectStore("encrypted_rcv_message_hashes"), {conn_id: connId, hash}) + return all.length > 0 + }) + }, + + // addEncryptedRcvMsgHash (from insertRcvMsgDetails_ AgentStore.hs:2641) + // SQL: INSERT INTO encrypted_rcv_message_hashes (conn_id, hash) VALUES (?,?) + async addEncryptedRcvMsgHash(connId, hash) { + return withTx("encrypted_rcv_message_hashes", "readwrite", async (tx) => { + await idbReq(tx.objectStore("encrypted_rcv_message_hashes").add({ + conn_id: connId, hash, created_at: new Date().toISOString(), + })) + }) + }, + } +} diff --git a/smp-web/src/agent/store.ts b/smp-web/src/agent/store.ts new file mode 100644 index 000000000..8aa65e4ed --- /dev/null +++ b/smp-web/src/agent/store.ts @@ -0,0 +1,279 @@ +// Agent store interface for IndexedDB. +// Each method mirrors a Haskell function in AgentStore.hs. +// Implementation in store-idb.ts. + +// -- Types matching Haskell store types + +export type ConnId = Uint8Array +export type UserId = number +export type EntityId = Uint8Array +export type InternalId = number +export type InternalRcvId = number +export type InternalSndId = number + +export type QueueStatus = "new" | "confirmed" | "secured" | "active" | "disabled" | "deleted" +export type ConnectionMode = "INV" | "CON" // SCMInvitation | SCMContact +export type RatchetSyncState = "ok" | "allowed" | "required" | "started" | "agreed" + +export interface ConnData { + connId: ConnId + connMode: ConnectionMode + userId: UserId + smpAgentVersion: number + enableNtfs: boolean + duplexHandshake: boolean + deleted: boolean + ratchetSyncState: RatchetSyncState + pqSupport: boolean + // Message ID counters + lastInternalMsgId: number + lastInternalRcvMsgId: number + lastInternalSndMsgId: number + lastExternalSndMsgId: number + lastRcvMsgHash: Uint8Array + lastSndMsgHash: Uint8Array +} + +export interface RcvQueue { + host: string + port: string + rcvId: Uint8Array + connId: ConnId + rcvPrivateKey: Uint8Array + rcvDhSecret: Uint8Array + e2ePrivKey: Uint8Array + e2eDhSecret: Uint8Array | null + sndId: Uint8Array + sndKey: Uint8Array | null + status: QueueStatus + smpClientVersion: number | null + dbQueueId: number + primary: boolean + replaceRcvQueueId: number | null + queueMode: string | null + serverKeyHash: Uint8Array | null + lastBrokerTs: string | null +} + +export interface SndQueue { + host: string + port: string + sndId: Uint8Array + connId: ConnId + sndPrivateKey: Uint8Array + e2eDhSecret: Uint8Array + status: QueueStatus + smpClientVersion: number + sndPublicKey: Uint8Array | null + e2ePubKey: Uint8Array | null + dbQueueId: number + primary: boolean + queueMode: string | null + serverKeyHash: Uint8Array | null +} + +export interface MsgMeta { + integrity: string // "OK" or error + recipient: [number, string] // (internalId, internalTs) + broker: [Uint8Array, string] // (brokerId/msgId, brokerTs) + sndMsgId: number + pqEncryption: boolean +} + +export interface RcvMsgData { + msgMeta: MsgMeta + msgType: string + msgFlags: number + msgBody: Uint8Array + internalRcvId: number + internalHash: Uint8Array + externalPrevSndHash: Uint8Array + encryptedMsgHash: Uint8Array +} + +export interface SndMsgData { + internalId: number + internalSndId: number + internalTs: string + msgType: string + msgFlags: number + msgBody: Uint8Array + pqEncryption: boolean + internalHash: Uint8Array + prevMsgHash: Uint8Array + msgEncryptKey: Uint8Array | null + paddedMsgLen: number | null + sndMessageBodyId: number | null +} + +export interface Confirmation { + confirmationId: Uint8Array + connId: ConnId + e2eSndPubKey: Uint8Array + senderKey: Uint8Array | null + ratchetState: Uint8Array + senderConnInfo: Uint8Array + accepted: boolean + ownConnInfo: Uint8Array | null + smpReplyQueues: Uint8Array | null // serialized + smpClientVersion: number | null +} + +export interface Invitation { + invitationId: Uint8Array + contactConnId: ConnId | null + crInvitation: Uint8Array + recipientConnInfo: Uint8Array + accepted: boolean + ownConnInfo: Uint8Array | null +} + +export interface RcvMsg { + internalId: number + msgMeta: MsgMeta + msgType: string + msgBody: Uint8Array + userAck: boolean + msgReceipt: Uint8Array | null +} + +export interface PendingQueueMsg { + connId: ConnId + sndQueueId: number + internalId: number + msgType: string + msgFlags: number + msgBody: Uint8Array + internalHash: Uint8Array + prevMsgHash: Uint8Array + pqEncryption: boolean + msgEncryptKey: Uint8Array | null + paddedMsgLen: number | null + sndMessageBodyId: number | null +} + +export interface AsyncCommand { + commandId: number + connId: ConnId + host: string | null + port: string | null + corrId: Uint8Array + commandTag: string + command: Uint8Array + agentVersion: number + serverKeyHash: Uint8Array | null + failed: boolean +} + +// -- Store interface +// Each method name matches the Haskell function in AgentStore.hs. + +export interface AgentStore { + // -- Users (AgentStore.hs:201-230) + createUserRecord(): Promise + getUserIds(): Promise + deleteUserRecord(userId: UserId): Promise + setUserDeleted(userId: UserId): Promise + + // -- Servers (AgentStore.hs:233-240) + createServer(host: string, port: string, keyHash: Uint8Array): Promise + + // -- Connections (AgentStore.hs:242-500) + createNewConn(connData: ConnData, connMode: ConnectionMode): Promise + getConn(connId: ConnId): Promise<{connData: ConnData, rcvQueues: RcvQueue[], sndQueues: SndQueue[]} | null> + getRcvConn(host: string, port: string, rcvId: Uint8Array): Promise<{connData: ConnData, rcvQueue: RcvQueue} | null> + getConnSubs(connIds: ConnId[]): Promise> + getConnsData(connIds: ConnId[]): Promise> + lockConnForUpdate(connId: ConnId): Promise // no-op in IndexedDB (single-threaded) + setConnDeleted(connId: ConnId, waitDelivery: boolean): Promise + setConnUserId(oldUserId: UserId, connId: ConnId, newUserId: UserId): Promise + setConnAgentVersion(connId: ConnId, version: number): Promise + setConnPQSupport(connId: ConnId, pqSupport: boolean): Promise + setConnRatchetSync(connId: ConnId, state: RatchetSyncState): Promise + updateNewConnJoin(connId: ConnId, agentVersion: number, pqSupport: boolean, enableNtfs: boolean): Promise + updateNewConnRcv(connId: ConnId, rcvQueue: RcvQueue, subMode: string): Promise + getDeletedConnIds(): Promise + getDeletedWaitingDeliveryConnIds(): Promise + getConnIds(): Promise + + // -- Queues (AgentStore.hs:500-700) + addConnRcvQueue(connId: ConnId, rcvQueue: RcvQueue, subMode: string): Promise + addConnSndQueue(connId: ConnId, sndQueue: SndQueue): Promise + setRcvQueueStatus(rcvQueue: RcvQueue, status: QueueStatus): Promise + setSndQueueStatus(sndQueue: SndQueue, status: QueueStatus): Promise + setRcvQueueConfirmedE2E(rcvQueue: RcvQueue, dhSecret: Uint8Array, smpClientVersion: number): Promise + setRcvQueuePrimary(connId: ConnId, rcvQueue: RcvQueue): Promise + deleteConnRcvQueue(rcvQueue: RcvQueue): Promise + deleteConnRecord(connId: ConnId): Promise + upgradeRcvConnToDuplex(connId: ConnId, sndQueue: SndQueue): Promise + upgradeSndConnToDuplex(connId: ConnId, rcvQueue: RcvQueue, subMode: string): Promise + getPrimaryRcvQueue(connId: ConnId): Promise + getRcvQueue(connId: ConnId, host: string, port: string, rcvId: Uint8Array): Promise + getDeletedRcvQueue(connId: ConnId, host: string, port: string, rcvId: Uint8Array): Promise + setConnectionNtfs(connId: ConnId, enable: boolean): Promise + + // -- Subscriptions (AgentStore.hs:700-800) + getSubscriptionServers(onlyNeeded: boolean): Promise> + getUserServerRcvQueueSubs(userId: UserId, host: string, port: string, onlyNeeded: boolean, batchSize: number, cursor: number | null): Promise<{queues: RcvQueue[], nextCursor: number | null}> + unsetQueuesToSubscribe(): Promise + getConnectionsForDelivery(): Promise + getAllSndQueuesForDelivery(): Promise + + // -- Confirmations (AgentStore.hs:800-870) + createConfirmation(confirmation: Confirmation): Promise + acceptConfirmation(confirmationId: Uint8Array, ownConnInfo: Uint8Array): Promise + getAcceptedConfirmation(connId: ConnId): Promise + removeConfirmations(connId: ConnId): Promise + + // -- Invitations (AgentStore.hs:870-920) + createInvitation(invitation: Invitation): Promise + getInvitation(invitationId: Uint8Array): Promise + acceptInvitation(invitationId: Uint8Array, ownConnInfo: Uint8Array): Promise + unacceptInvitation(invitationId: Uint8Array): Promise + deleteInvitation(invitationId: Uint8Array): Promise + + // -- Messages (AgentStore.hs:873-1050) + updateRcvIds(connId: ConnId): Promise<{internalId: number, internalRcvId: number, prevExternalSndId: number, prevRcvMsgHash: Uint8Array}> + createRcvMsg(connId: ConnId, rcvQueue: RcvQueue, rcvMsgData: RcvMsgData): Promise + setLastBrokerTs(connId: ConnId, dbQueueId: number, brokerTs: string): Promise + updateRcvMsgHash(connId: ConnId, sndMsgId: number, internalRcvId: number, hash: Uint8Array): Promise + createSndMsgBody(agentMsg: Uint8Array): Promise + updateSndIds(connId: ConnId): Promise<{internalId: number, internalSndId: number, prevSndMsgHash: Uint8Array}> + createSndMsg(connId: ConnId, sndMsgData: SndMsgData): Promise + updateSndMsgHash(connId: ConnId, internalSndId: number, hash: Uint8Array): Promise + createSndMsgDelivery(connId: ConnId, sndQueue: SndQueue, internalId: number): Promise + getPendingQueueMsg(connId: ConnId, sndQueue: SndQueue): Promise + updatePendingMsgRIState(connId: ConnId, msgId: number, retryIntSlow: number | null, retryIntFast: number | null): Promise + setMsgUserAck(connId: ConnId, internalId: number): Promise<{rcvQueue: RcvQueue, brokerId: Uint8Array}> + getRcvMsg(connId: ConnId, internalId: number): Promise + getLastMsg(connId: ConnId, brokerId: Uint8Array): Promise + incMsgRcvAttempts(connId: ConnId, internalId: number): Promise + checkRcvMsgHashExists(connId: ConnId, hash: Uint8Array): Promise + getRcvMsgBrokerTs(connId: ConnId, brokerId: Uint8Array): Promise + deleteMsg(connId: ConnId, internalId: number): Promise + deleteDeliveredSndMsg(connId: ConnId, internalSndId: number): Promise + deleteSndMsgDelivery(connId: ConnId, sndQueue: SndQueue, msgId: number, keepForReceipt: boolean): Promise + getSndMsgViaRcpt(connId: ConnId, sndMsgId: number): Promise<{internalId: number, msgType: string, internalHash: Uint8Array, msgReceipt: Uint8Array | null} | null> + updateSndMsgRcpt(connId: ConnId, sndMsgId: number, receipt: Uint8Array): Promise + + // -- Ratchet (AgentStore.hs:1300-1400) + createRatchetX3dhKeys(connId: ConnId, privKey1: Uint8Array, privKey2: Uint8Array, pqKem: Uint8Array | null): Promise + getRatchetX3dhKeys(connId: ConnId): Promise<{privKey1: Uint8Array, privKey2: Uint8Array, pqKem: Uint8Array | null} | null> + createRatchet(connId: ConnId, ratchetState: Uint8Array): Promise + getRatchet(connId: ConnId): Promise + getRatchetForUpdate(connId: ConnId): Promise // same as getRatchet in IndexedDB (single-threaded) + getSkippedMsgKeys(connId: ConnId): Promise>> + updateRatchet(connId: ConnId, ratchetState: Uint8Array, skippedMsgDiff: unknown): Promise + + // -- Commands (AgentStore.hs:1400-1480) + createCommand(corrId: Uint8Array, connId: ConnId, host: string | null, port: string | null, command: AsyncCommand): Promise + getPendingCommandServers(connIds: ConnId[]): Promise> + getAllPendingCommandConns(): Promise> + getPendingServerCommand(host: string, port: string): Promise + updateCommandServer(commandId: number, host: string, port: string): Promise + deleteCommand(commandId: number): Promise + + // -- Encrypted message hash dedup (AgentStore.hs:1200-1220) + checkRcvMsgHashExists_encrypted(connId: ConnId, hash: Uint8Array): Promise + addEncryptedRcvMsgHash(connId: ConnId, hash: Uint8Array): Promise +} diff --git a/smp-web/src/idb-keys.d.ts b/smp-web/src/idb-keys.d.ts new file mode 100644 index 000000000..7e0e8845e --- /dev/null +++ b/smp-web/src/idb-keys.d.ts @@ -0,0 +1,15 @@ +// Override IDBValidKey to accept Uint8Array (supported in modern browsers) +interface IDBObjectStore { + get(query: any): IDBRequest + getAll(query?: any, count?: number): IDBRequest + getAllKeys(query?: any, count?: number): IDBRequest + add(value: any, key?: any): IDBRequest + put(value: any, key?: any): IDBRequest + delete(query: any): IDBRequest +} + +interface IDBIndex { + get(query: any): IDBRequest + getAll(query?: any, count?: number): IDBRequest + getAllKeys(query?: any, count?: number): IDBRequest +} diff --git a/smp-web/tests/store-test.ts b/smp-web/tests/store-test.ts new file mode 100644 index 000000000..d97631e29 --- /dev/null +++ b/smp-web/tests/store-test.ts @@ -0,0 +1,818 @@ +// Agent store scenario tests using fake-indexeddb. +// Each scenario exercises a full lifecycle: create → update → read → delete → verify gone. +// Every store method is called from at least one test. + +import "fake-indexeddb/auto" +import {openAgentStore} from "../dist/agent/store-idb.js" +import type {AgentStore} from "../dist/agent/store.js" + +// The store returns raw IndexedDB rows with snake_case field names. +// The TypeScript interface types use camelCase but the underlying data is snake_case. +// We use `any` casts in assertions to access the actual field names. +type Row = any + +// -- Helpers + +function bytes(hex: string): Uint8Array { + const b = new Uint8Array(hex.length / 2) + for (let i = 0; i < hex.length; i += 2) b[i / 2] = parseInt(hex.slice(i, i + 2), 16) + return b +} + +function hex(b: Uint8Array): string { + return Array.from(b, x => x.toString(16).padStart(2, "0")).join("") +} + +function randomBytes(n: number): Uint8Array { + const b = new Uint8Array(n) + for (let i = 0; i < n; i++) b[i] = Math.floor(Math.random() * 256) + return b +} + +let passed = 0 +let failed = 0 + +function assert(cond: boolean, msg: string) { + if (!cond) { + console.error("FAIL:", msg) + failed++ + } else { + passed++ + } +} + +function assertEq(a: any, b: any, msg: string) { + const av = a instanceof Uint8Array ? hex(a) : JSON.stringify(a) + const bv = b instanceof Uint8Array ? hex(b) : JSON.stringify(b) + assert(av === bv, `${msg}: expected ${bv}, got ${av}`) +} + +async function assertThrows(fn: () => Promise, msg: string) { + try { + await fn() + assert(false, `${msg}: expected throw`) + } catch { + passed++ + } +} + +// -- Scenarios + +async function testUsers(store: AgentStore) { + console.log(" users...") + // createUserRecord, getUserIds + const uid1 = await store.createUserRecord() + const uid2 = await store.createUserRecord() + let ids = await store.getUserIds() + assert(ids.includes(uid1) && ids.includes(uid2), "getUserIds returns both users") + + // setUserDeleted — marks user as deleted, getUserIds should exclude it + await store.setUserDeleted(uid1) + ids = await store.getUserIds() + assert(!ids.includes(uid1), "deleted user not in getUserIds") + assert(ids.includes(uid2), "non-deleted user still in getUserIds") + + // deleteUserRecord — hard delete + await store.deleteUserRecord(uid2) + ids = await store.getUserIds() + assert(!ids.includes(uid2), "hard-deleted user gone") +} + +async function testServers(store: AgentStore) { + console.log(" servers...") + // createServer — insert or ignore + const kh = randomBytes(32) + await store.createServer("smp1.example.com", "5223", kh) + // Duplicate should not throw + await store.createServer("smp1.example.com", "5223", kh) +} + +async function testConnectionsAndQueues(store: AgentStore) { + console.log(" connections and queues...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + const connId2 = randomBytes(24) + + // createNewConn + const created = await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: true, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + assertEq(created, connId, "createNewConn returns connId") + + // getConn — should find it + const got = await store.getConn(connId) + assert(got !== null, "getConn finds connection") + assertEq((got!.connData as Row).conn_id, connId, "getConn connId matches") + + // getConnIds + const allIds = await store.getConnIds() + assert(allIds.some(id => hex(id) === hex(connId)), "getConnIds includes new conn") + + // setConnAgentVersion + await store.setConnAgentVersion(connId, 8) + const got2 = await store.getConn(connId) + assertEq((got2!.connData as Row).smp_agent_version, 8, "setConnAgentVersion updated") + + // setConnPQSupport + await store.setConnPQSupport(connId, false) + const got3 = await store.getConn(connId) + assertEq((got3!.connData as Row).pq_support, 0, "setConnPQSupport updated") + + // setConnRatchetSync + await store.setConnRatchetSync(connId, "required") + const got4 = await store.getConn(connId) + assertEq((got4!.connData as Row).ratchet_sync_state, "required", "setConnRatchetSync updated") + + // setConnectionNtfs + await store.setConnectionNtfs(connId, false) + const got5 = await store.getConn(connId) + assertEq((got5!.connData as Row).enable_ntfs, 0, "setConnectionNtfs updated") + + // lockConnForUpdate — no-op, should not throw + await store.lockConnForUpdate(connId) + + // addConnRcvQueue + const rcvQ = await store.addConnRcvQueue(connId, { + host: "smp1.example.com", port: "5223", rcvId: randomBytes(24), + connId, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "new" as const, + smpClientVersion: 7, dbQueueId: 0, primary: true, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMSubscribe") + assert(rcvQ.dbQueueId >= 1, "addConnRcvQueue assigns dbQueueId") + + // getPrimaryRcvQueue + const primary = await store.getPrimaryRcvQueue(connId) + assert(primary !== null, "getPrimaryRcvQueue finds queue") + assertEq((primary as Row).rcv_primary, 1, "primary queue is marked primary") + + // getRcvConn + const rcvConn = await store.getRcvConn(rcvQ.host, rcvQ.port, rcvQ.rcvId) + assert(rcvConn !== null, "getRcvConn finds by host/port/rcvId") + + // getRcvQueue + const rq = await store.getRcvQueue(connId, rcvQ.host, rcvQ.port, rcvQ.rcvId) + assert(rq !== null, "getRcvQueue finds queue") + + // setRcvQueueStatus + await store.setRcvQueueStatus(rcvQ, "confirmed") + const rq2 = await store.getRcvQueue(connId, rcvQ.host, rcvQ.port, rcvQ.rcvId) + assertEq(rq2!.status, "confirmed", "setRcvQueueStatus updated") + + // setRcvQueueConfirmedE2E + const dhSecret = randomBytes(32) + await store.setRcvQueueConfirmedE2E(rcvQ, dhSecret, 7) + const rq3 = await store.getRcvQueue(connId, rcvQ.host, rcvQ.port, rcvQ.rcvId) + assertEq((rq3 as Row).e2e_dh_secret, dhSecret, "setRcvQueueConfirmedE2E updated dh secret") + assertEq((rq3 as Row).status, "confirmed", "setRcvQueueConfirmedE2E sets confirmed") + + // addConnSndQueue + upgradeRcvConnToDuplex (same operation) + const sndQ = { + host: "smp2.example.com", port: "5223", sndId: randomBytes(24), + connId, sndPrivateKey: randomBytes(32), e2eDhSecret: randomBytes(32), + status: "confirmed" as const, smpClientVersion: 7, + sndPublicKey: randomBytes(32), e2ePubKey: randomBytes(32), + dbQueueId: 0, primary: true, queueMode: null, serverKeyHash: randomBytes(32), + } + await store.upgradeRcvConnToDuplex(connId, sndQ) + const gotDuplex = await store.getConn(connId) + assert(gotDuplex!.sndQueues.length >= 1, "upgradeRcvConnToDuplex added snd queue") + + // setSndQueueStatus + await store.setSndQueueStatus(sndQ, "active") + + // getConnSubs, getConnsData + const subs = await store.getConnSubs([connId]) + assert(subs.size === 1, "getConnSubs returns 1 entry") + const connsData = await store.getConnsData([connId]) + assert(connsData.size === 1, "getConnsData returns 1 entry") + + // Create second connection for setConnUserId + await store.createNewConn({ + connId: connId2, connMode: "CON", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "CON") + const userId2 = await store.createUserRecord() + await store.setConnUserId(userId, connId2, userId2) + const got6 = await store.getConn(connId2) + assertEq((got6!.connData as Row).user_id, userId2, "setConnUserId updated") + + // updateNewConnJoin + await store.updateNewConnJoin(connId, 9, true, false) + const got7 = await store.getConn(connId) + assertEq((got7!.connData as Row).smp_agent_version, 9, "updateNewConnJoin updated version") + + // updateNewConnRcv — adds a rcv queue (same as addConnRcvQueue) + const rcvQ2 = await store.updateNewConnRcv(connId2, { + host: "smp3.example.com", port: "5223", rcvId: randomBytes(24), + connId: connId2, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "new" as const, + smpClientVersion: 7, dbQueueId: 0, primary: true, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMSubscribe") + assert(rcvQ2.dbQueueId >= 1, "updateNewConnRcv assigns dbQueueId") + + // upgradeSndConnToDuplex — add rcv queue to a snd-only connection + const connId3 = randomBytes(24) + await store.createNewConn({ + connId: connId3, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + await store.addConnSndQueue(connId3, { + host: "smp4.example.com", port: "5223", sndId: randomBytes(24), + connId: connId3, sndPrivateKey: randomBytes(32), e2eDhSecret: randomBytes(32), + status: "confirmed" as const, smpClientVersion: 7, + sndPublicKey: null, e2ePubKey: null, dbQueueId: 0, primary: true, + queueMode: null, serverKeyHash: randomBytes(32), + }) + const rcvQ3 = await store.upgradeSndConnToDuplex(connId3, { + host: "smp4.example.com", port: "5223", rcvId: randomBytes(24), + connId: connId3, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "new" as const, + smpClientVersion: 7, dbQueueId: 0, primary: true, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMSubscribe") + assert(rcvQ3.dbQueueId >= 1, "upgradeSndConnToDuplex added rcv queue") + + // setRcvQueuePrimary — add second rcv queue, make it primary + const rcvQ4 = await store.addConnRcvQueue(connId, { + host: "smp5.example.com", port: "5223", rcvId: randomBytes(24), + connId, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "new" as const, + smpClientVersion: 7, dbQueueId: 0, primary: false, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMSubscribe") + await store.setRcvQueuePrimary(connId, rcvQ4) + const newPrimary = await store.getPrimaryRcvQueue(connId) + assertEq((newPrimary as Row).rcv_queue_id, rcvQ4.dbQueueId, "setRcvQueuePrimary changed primary") + + // getDeletedRcvQueue — first delete a queue, then find it + // deleteConnRcvQueue physically deletes, so getDeletedRcvQueue won't find it + // We need to test the soft-delete path — but deleteConnRcvQueue does hard delete + // Just verify it returns null for non-deleted + const drq = await store.getDeletedRcvQueue(connId, rcvQ.host, rcvQ.port, rcvQ.rcvId) + assert(drq === null, "getDeletedRcvQueue returns null for non-deleted queue") + + // deleteConnRcvQueue + await store.deleteConnRcvQueue(rcvQ4) + const afterDel = await store.getRcvQueue(connId, rcvQ4.host, rcvQ4.port, rcvQ4.rcvId) + assert(afterDel === null, "deleteConnRcvQueue removes queue") + + // setConnDeleted (waitDelivery=true) + await store.setConnDeleted(connId2, true) + const waitDel = await store.getDeletedWaitingDeliveryConnIds() + assert(waitDel.some(id => hex(id) === hex(connId2)), "setConnDeleted waitDelivery appears in getDeletedWaitingDeliveryConnIds") + + // setConnDeleted (waitDelivery=false) + await store.setConnDeleted(connId3, false) + const delIds = await store.getDeletedConnIds() + assert(delIds.some(id => hex(id) === hex(connId3)), "setConnDeleted appears in getDeletedConnIds") + + // deleteConnRecord + await store.deleteConnRecord(connId3) + const gone = await store.getConn(connId3) + assert(gone === null, "deleteConnRecord removes connection") +} + +async function testSubscriptions(store: AgentStore) { + console.log(" subscriptions...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + await store.createServer("sub.example.com", "5223", randomBytes(32)) + const rcvQ = await store.addConnRcvQueue(connId, { + host: "sub.example.com", port: "5223", rcvId: randomBytes(24), + connId, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "new" as const, + smpClientVersion: 7, dbQueueId: 0, primary: true, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMOnlyCreate") + + // getSubscriptionServers — onlyNeeded=true should find our to_subscribe=1 queue + const srvs = await store.getSubscriptionServers(true) + assert(srvs.some(s => s.host === "sub.example.com"), "getSubscriptionServers finds queue with to_subscribe") + + // getSubscriptionServers — onlyNeeded=false + const allSrvs = await store.getSubscriptionServers(false) + assert(allSrvs.some(s => s.host === "sub.example.com"), "getSubscriptionServers(false) finds all") + + // getUserServerRcvQueueSubs + const {queues} = await store.getUserServerRcvQueueSubs(userId, "sub.example.com", "5223", true, 10, null) + assert(queues.length >= 1, "getUserServerRcvQueueSubs finds queues") + + // unsetQueuesToSubscribe + await store.unsetQueuesToSubscribe() + const srvsAfter = await store.getSubscriptionServers(true) + assert(!srvsAfter.some(s => s.host === "sub.example.com"), "unsetQueuesToSubscribe clears to_subscribe") + + // getConnectionsForDelivery, getAllSndQueuesForDelivery — need snd deliveries + // Add snd queue and delivery + const sndQ = { + host: "sub.example.com", port: "5223", sndId: randomBytes(24), + connId, sndPrivateKey: randomBytes(32), e2eDhSecret: randomBytes(32), + status: "active" as const, smpClientVersion: 7, + sndPublicKey: null, e2ePubKey: null, dbQueueId: 0, primary: true, + queueMode: null, serverKeyHash: randomBytes(32), + } + await store.addConnSndQueue(connId, sndQ) + // Need to create a message first + const {internalId, internalSndId, prevSndMsgHash} = await store.updateSndIds(connId) + await store.createSndMsg(connId, { + internalId, internalSndId, internalTs: new Date().toISOString(), + msgType: "HELLO", msgFlags: 0, msgBody: randomBytes(10), + pqEncryption: false, internalHash: randomBytes(32), + prevMsgHash: prevSndMsgHash, msgEncryptKey: null, paddedMsgLen: null, sndMessageBodyId: null, + }) + // Get the snd queue with its actual dbQueueId + const gotConn = await store.getConn(connId) + const actualSndQ = gotConn!.sndQueues[0] as Row + // Raw IDB row has snd_queue_id, interface expects dbQueueId + await store.createSndMsgDelivery(connId, {dbQueueId: actualSndQ.snd_queue_id} as any, internalId) + + const deliveryConns = await store.getConnectionsForDelivery() + assert(deliveryConns.some(id => hex(id) === hex(connId)), "getConnectionsForDelivery finds conn with delivery") + + const deliverySndQs = await store.getAllSndQueuesForDelivery() + assert(deliverySndQs.length >= 1, "getAllSndQueuesForDelivery finds queues") +} + +async function testConfirmations(store: AgentStore) { + console.log(" confirmations...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + + const confId = randomBytes(16) + // createConfirmation + const returned = await store.createConfirmation({ + confirmationId: confId, connId, + e2eSndPubKey: randomBytes(32), senderKey: randomBytes(32), + ratchetState: randomBytes(64), senderConnInfo: randomBytes(50), + accepted: false, ownConnInfo: null, + smpReplyQueues: randomBytes(100), smpClientVersion: 7, + }) + assertEq(returned, confId, "createConfirmation returns confirmationId") + + // getAcceptedConfirmation — not accepted yet + const notAccepted = await store.getAcceptedConfirmation(connId) + assert(notAccepted === null, "getAcceptedConfirmation returns null before acceptance") + + // acceptConfirmation + const ownInfo = randomBytes(40) + const accepted = await store.acceptConfirmation(confId, ownInfo) + assert(accepted !== null, "acceptConfirmation returns confirmation") + assertEq(accepted.accepted, 1, "acceptConfirmation sets accepted=1") + + // getAcceptedConfirmation — now accepted + const gotAccepted = await store.getAcceptedConfirmation(connId) + assert(gotAccepted !== null, "getAcceptedConfirmation finds accepted confirmation") + assertEq((gotAccepted as Row).own_conn_info, ownInfo, "accepted confirmation has ownConnInfo") + + // removeConfirmations + await store.removeConfirmations(connId) + const afterRemove = await store.getAcceptedConfirmation(connId) + assert(afterRemove === null, "removeConfirmations deletes all confirmations for conn") +} + +async function testInvitations(store: AgentStore) { + console.log(" invitations...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "CON", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "CON") + + const invId = randomBytes(16) + // createInvitation + const returned = await store.createInvitation({ + invitationId: invId, contactConnId: connId, + crInvitation: randomBytes(200), recipientConnInfo: randomBytes(50), + accepted: false, ownConnInfo: null, + }) + assertEq(returned, invId, "createInvitation returns invitationId") + + // getInvitation — not accepted, should find + const got = await store.getInvitation(invId) + assert(got !== null, "getInvitation finds unaccepted invitation") + + // acceptInvitation + const ownInfo = randomBytes(40) + await store.acceptInvitation(invId, ownInfo) + // getInvitation — accepted, should NOT find (WHERE accepted = 0) + const gotAfterAccept = await store.getInvitation(invId) + assert(gotAfterAccept === null, "getInvitation returns null for accepted invitation") + + // unacceptInvitation + await store.unacceptInvitation(invId) + const gotAfterUnaccept = await store.getInvitation(invId) + assert(gotAfterUnaccept !== null, "unacceptInvitation resets accepted to 0") + assert((gotAfterUnaccept as Row).own_conn_info === null, "unacceptInvitation clears ownConnInfo") + + // deleteInvitation + await store.deleteInvitation(invId) + const gotAfterDelete = await store.getInvitation(invId) + assert(gotAfterDelete === null, "deleteInvitation removes invitation") +} + +async function testReceiveMessages(store: AgentStore) { + console.log(" receive messages...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + await store.createServer("rcv.example.com", "5223", randomBytes(32)) + const rcvQ = await store.addConnRcvQueue(connId, { + host: "rcv.example.com", port: "5223", rcvId: randomBytes(24), + connId, rcvPrivateKey: randomBytes(32), rcvDhSecret: randomBytes(32), + e2ePrivKey: randomBytes(32), e2eDhSecret: null, + sndId: randomBytes(24), sndKey: null, status: "active" as const, + smpClientVersion: 7, dbQueueId: 0, primary: true, + replaceRcvQueueId: null, queueMode: null, + serverKeyHash: randomBytes(32), lastBrokerTs: null, + }, "SMSubscribe") + + // updateRcvIds + const {internalId, internalRcvId, prevExternalSndId, prevRcvMsgHash} = await store.updateRcvIds(connId) + assertEq(internalId, 1, "updateRcvIds first internalId=1") + assertEq(internalRcvId, 1, "updateRcvIds first internalRcvId=1") + assertEq(prevExternalSndId, 0, "updateRcvIds prevExternalSndId=0") + + const brokerId = randomBytes(24) + const brokerTs = new Date().toISOString() + const internalHash = randomBytes(32) + const encryptedMsgHash = randomBytes(32) + const msgBody = randomBytes(100) + + // createRcvMsg — exercises insertRcvMsgBase_, insertRcvMsgDetails_, updateRcvMsgHash, setLastBrokerTs + await store.createRcvMsg(connId, rcvQ, { + msgMeta: { + integrity: "OK", + recipient: [internalId, new Date().toISOString()], + broker: [brokerId, brokerTs], + sndMsgId: 1, + pqEncryption: false, + }, + msgType: "MSG", + msgFlags: 0, + msgBody, + internalRcvId, + internalHash, + externalPrevSndHash: randomBytes(32), + encryptedMsgHash, + }) + + // getRcvMsg + const rcvMsg = await store.getRcvMsg(connId, internalId) + assert(rcvMsg !== null, "getRcvMsg finds message") + assertEq(rcvMsg!.msgType, "MSG", "getRcvMsg msgType matches") + + // getLastMsg — verify the msg is "last" (conn.last_internal_msg_id matches) + const lastMsg = await store.getLastMsg(connId, brokerId) + assert(lastMsg !== null, "getLastMsg finds message by brokerId") + + // getRcvMsgBrokerTs + const ts = await store.getRcvMsgBrokerTs(connId, brokerId) + assert(ts !== null, "getRcvMsgBrokerTs finds broker ts") + + // checkRcvMsgHashExists — encrypted hash was inserted by createRcvMsg + const hashExists = await store.checkRcvMsgHashExists(connId, encryptedMsgHash) + assert(hashExists, "checkRcvMsgHashExists finds hash inserted by createRcvMsg") + + // incMsgRcvAttempts + const attempts = await store.incMsgRcvAttempts(connId, internalId) + assertEq(attempts, 1, "incMsgRcvAttempts returns 1 after first increment") + const attempts2 = await store.incMsgRcvAttempts(connId, internalId) + assertEq(attempts2, 2, "incMsgRcvAttempts returns 2 after second increment") + + // setMsgUserAck + const {rcvQueue: ackQ, brokerId: ackBrokerId} = await store.setMsgUserAck(connId, internalId) + assert(ackQ !== null, "setMsgUserAck returns rcvQueue") + assertEq(ackBrokerId, brokerId, "setMsgUserAck returns correct brokerId") + + // Verify user_ack was set + const rcvMsgAfterAck = await store.getRcvMsg(connId, internalId) + assert(rcvMsgAfterAck!.userAck, "setMsgUserAck sets userAck=true") + + // setLastBrokerTs — standalone call + const newTs = new Date().toISOString() + await store.setLastBrokerTs(connId, rcvQ.dbQueueId, newTs) + + // updateRcvMsgHash — standalone call + await store.updateRcvMsgHash(connId, 2, internalRcvId, randomBytes(32)) + + // deleteMsg + await store.deleteMsg(connId, internalId) + const deletedMsg = await store.getRcvMsg(connId, internalId) + assert(deletedMsg === null, "deleteMsg removes message") +} + +async function testSendMessages(store: AgentStore) { + console.log(" send messages...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + await store.createServer("snd.example.com", "5223", randomBytes(32)) + await store.addConnSndQueue(connId, { + host: "snd.example.com", port: "5223", sndId: randomBytes(24), + connId, sndPrivateKey: randomBytes(32), e2eDhSecret: randomBytes(32), + status: "active" as const, smpClientVersion: 7, + sndPublicKey: null, e2ePubKey: null, dbQueueId: 0, primary: true, + queueMode: null, serverKeyHash: randomBytes(32), + }) + const gotConn = await store.getConn(connId) + const sndQueue = gotConn!.sndQueues[0] + + // createSndMsgBody + const agentMsg = randomBytes(200) + const bodyId = await store.createSndMsgBody(agentMsg) + assert(bodyId >= 1, "createSndMsgBody returns positive id") + + // updateSndIds + const {internalId, internalSndId, prevSndMsgHash} = await store.updateSndIds(connId) + assertEq(internalId, 1, "updateSndIds first internalId=1") + assertEq(internalSndId, 1, "updateSndIds first internalSndId=1") + + const internalHash = randomBytes(32) + + // createSndMsg + await store.createSndMsg(connId, { + internalId, internalSndId, internalTs: new Date().toISOString(), + msgType: "SEND", msgFlags: 0, msgBody: randomBytes(100), + pqEncryption: false, internalHash, + prevMsgHash: prevSndMsgHash, msgEncryptKey: randomBytes(32), + paddedMsgLen: 16384, sndMessageBodyId: bodyId, + }) + + // updateSndMsgHash — standalone + await store.updateSndMsgHash(connId, internalSndId, internalHash) + + // createSndMsgDelivery + await store.createSndMsgDelivery(connId, sndQueue, internalId) + + // getPendingQueueMsg + const pending = await store.getPendingQueueMsg(connId, sndQueue) + assert(pending !== null, "getPendingQueueMsg finds pending message") + assertEq(pending!.msgType, "SEND", "getPendingQueueMsg msgType matches") + assertEq(pending!.internalId, internalId, "getPendingQueueMsg internalId matches") + + // updatePendingMsgRIState + await store.updatePendingMsgRIState(connId, internalId, 30, 5) + + // getSndMsgViaRcpt + const sndMsg = await store.getSndMsgViaRcpt(connId, internalSndId) + assert(sndMsg !== null, "getSndMsgViaRcpt finds message") + assertEq(sndMsg!.internalId, internalId, "getSndMsgViaRcpt internalId matches") + assertEq(sndMsg!.internalHash, internalHash, "getSndMsgViaRcpt hash matches") + + // updateSndMsgRcpt + const receipt = randomBytes(8) + await store.updateSndMsgRcpt(connId, internalSndId, receipt) + + // deleteSndMsgDelivery — deletes delivery, then msg if no more deliveries + await store.deleteSndMsgDelivery(connId, sndQueue, internalId, false) + + // Create a second message for deleteDeliveredSndMsg + const ids2 = await store.updateSndIds(connId) + await store.createSndMsg(connId, { + internalId: ids2.internalId, internalSndId: ids2.internalSndId, + internalTs: new Date().toISOString(), + msgType: "SEND", msgFlags: 0, msgBody: randomBytes(50), + pqEncryption: false, internalHash: randomBytes(32), + prevMsgHash: ids2.prevSndMsgHash, msgEncryptKey: null, paddedMsgLen: null, sndMessageBodyId: null, + }) + + // deleteDeliveredSndMsg — no deliveries exist, so should delete msg + await store.deleteDeliveredSndMsg(connId, ids2.internalId) +} + +async function testRatchet(store: AgentStore) { + console.log(" ratchet...") + const connId = randomBytes(24) + const userId = await store.createUserRecord() + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: true, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + + const privKey1 = randomBytes(56) + const privKey2 = randomBytes(56) + const pqKem = randomBytes(100) + + // createRatchetX3dhKeys + await store.createRatchetX3dhKeys(connId, privKey1, privKey2, pqKem) + + // getRatchetX3dhKeys + const keys = await store.getRatchetX3dhKeys(connId) + assert(keys !== null, "getRatchetX3dhKeys finds keys") + assertEq(keys!.privKey1, privKey1, "x3dh privKey1 matches") + assertEq(keys!.privKey2, privKey2, "x3dh privKey2 matches") + assertEq(keys!.pqKem, pqKem, "x3dh pqKem matches") + + // getRatchet — no ratchet state yet + const noRatchet = await store.getRatchet(connId) + assert(noRatchet === null, "getRatchet returns null before createRatchet") + + // createRatchet — upserts, clearing x3dh keys + const ratchetState = randomBytes(200) + await store.createRatchet(connId, ratchetState) + + // getRatchet + const gotRatchet = await store.getRatchet(connId) + assertEq(gotRatchet, ratchetState, "getRatchet returns stored state") + + // getRatchetForUpdate — same as getRatchet in IndexedDB + const gotForUpdate = await store.getRatchetForUpdate(connId) + assertEq(gotForUpdate, ratchetState, "getRatchetForUpdate returns stored state") + + // x3dh keys should be cleared after createRatchet + const keysAfter = await store.getRatchetX3dhKeys(connId) + assert(keysAfter === null, "createRatchet clears x3dh keys") + + // getSkippedMsgKeys — empty initially + const noSkipped = await store.getSkippedMsgKeys(connId) + assertEq(noSkipped.size, 0, "getSkippedMsgKeys empty initially") + + // updateRatchet with SMDAdd + const headerKey = randomBytes(32) + const msgKey = randomBytes(32) + const newRatchetState = randomBytes(200) + const addKeys = new Map>() + const inner = new Map() + inner.set(0, msgKey) + inner.set(1, randomBytes(32)) + addKeys.set(headerKey, inner) + await store.updateRatchet(connId, newRatchetState, {type: "add", keys: addKeys}) + + // Verify ratchet state updated + const updatedRatchet = await store.getRatchet(connId) + assertEq(updatedRatchet, newRatchetState, "updateRatchet updates state") + + // Verify skipped keys added + const skipped = await store.getSkippedMsgKeys(connId) + assert(skipped.size >= 1, "updateRatchet SMDAdd adds skipped keys") + + // updateRatchet with SMDRemove + const hkHex = Array.from(headerKey, x => x.toString(16).padStart(2, "0")).join("") + await store.updateRatchet(connId, randomBytes(200), {type: "remove", headerKey, msgN: 0}) + const afterRemove = await store.getSkippedMsgKeys(connId) + // Should have 1 key remaining (msgN=1) instead of 2 + let totalKeys = 0 + for (const [, m] of afterRemove) totalKeys += m.size + assertEq(totalKeys, 1, "updateRatchet SMDRemove removes specific key") + + // updateRatchet with noChange + const stateBeforeNoChange = randomBytes(200) + await store.updateRatchet(connId, stateBeforeNoChange, {type: "noChange"}) + const afterNoChange = await store.getRatchet(connId) + assertEq(afterNoChange, stateBeforeNoChange, "updateRatchet SMDNoChange only updates state") +} + +async function testCommands(store: AgentStore) { + console.log(" commands...") + const userId = await store.createUserRecord() + const connId = randomBytes(24) + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + + const corrId = randomBytes(24) + // createCommand + const cmdId = await store.createCommand(corrId, connId, "cmd.example.com", "5223", { + commandId: 0, connId, host: "cmd.example.com", port: "5223", + corrId, commandTag: "NEW", command: randomBytes(50), + agentVersion: 7, serverKeyHash: randomBytes(32), failed: false, + }) + assert(cmdId >= 1, "createCommand returns positive id") + + // getPendingCommandServers + const servers = await store.getPendingCommandServers([connId]) + assert(servers.some(s => s.host === "cmd.example.com"), "getPendingCommandServers finds command server") + + // getAllPendingCommandConns + const allConns = await store.getAllPendingCommandConns() + assert(allConns.some(c => hex(c.connId) === hex(connId)), "getAllPendingCommandConns finds conn") + + // getPendingServerCommand + const pendingCmd = await store.getPendingServerCommand("cmd.example.com", "5223") + assert(pendingCmd !== null, "getPendingServerCommand finds command") + + // updateCommandServer + await store.updateCommandServer(cmdId, "cmd2.example.com", "5224") + const updated = await store.getPendingServerCommand("cmd2.example.com", "5224") + assert(updated !== null, "updateCommandServer changes server") + const oldHost = await store.getPendingServerCommand("cmd.example.com", "5223") + assert(oldHost === null, "updateCommandServer — old host returns nothing") + + // deleteCommand + await store.deleteCommand(cmdId) + const deleted = await store.getPendingServerCommand("cmd2.example.com", "5224") + assert(deleted === null, "deleteCommand removes command") +} + +async function testHashDedup(store: AgentStore) { + console.log(" hash dedup...") + const connId = randomBytes(24) + const userId = await store.createUserRecord() + await store.createNewConn({ + connId, connMode: "INV", userId, smpAgentVersion: 7, + enableNtfs: true, duplexHandshake: true, deleted: false, + ratchetSyncState: "ok", pqSupport: false, + lastInternalMsgId: 0, lastInternalRcvMsgId: 0, lastInternalSndMsgId: 0, + lastExternalSndMsgId: 0, lastRcvMsgHash: new Uint8Array(0), lastSndMsgHash: new Uint8Array(0), + }, "INV") + + const hash = randomBytes(32) + + // checkRcvMsgHashExists_encrypted — not yet + const before = await store.checkRcvMsgHashExists_encrypted(connId, hash) + assert(!before, "checkRcvMsgHashExists_encrypted returns false before add") + + // addEncryptedRcvMsgHash + await store.addEncryptedRcvMsgHash(connId, hash) + + // checkRcvMsgHashExists_encrypted — now exists + const after = await store.checkRcvMsgHashExists_encrypted(connId, hash) + assert(after, "checkRcvMsgHashExists_encrypted returns true after add") + + // Different hash should not exist + const other = await store.checkRcvMsgHashExists_encrypted(connId, randomBytes(32)) + assert(!other, "checkRcvMsgHashExists_encrypted returns false for different hash") +} + +// -- Run all + +async function main() { + console.log("Agent store tests") + const store = await openAgentStore() + + await testUsers(store) + await testServers(store) + await testConnectionsAndQueues(store) + await testSubscriptions(store) + await testConfirmations(store) + await testInvitations(store) + await testReceiveMessages(store) + await testSendMessages(store) + await testRatchet(store) + await testCommands(store) + await testHashDedup(store) + + console.log(`\n${passed} passed, ${failed} failed`) + if (failed > 0) process.exit(1) +} + +main().catch(e => { console.error("FATAL:", e?.message || e, e?.stack); process.exit(1) }) diff --git a/smp-web/tsconfig.json b/smp-web/tsconfig.json index 2d66241bd..b2d62ce39 100644 --- a/smp-web/tsconfig.json +++ b/smp-web/tsconfig.json @@ -3,7 +3,7 @@ "target": "ES2022", "module": "ES2022", "moduleResolution": "node", - "lib": ["ES2022"], + "lib": ["ES2022", "DOM"], "outDir": "dist", "rootDir": "src", "declaration": true, diff --git a/smp-web/tsconfig.test.json b/smp-web/tsconfig.test.json index 960f58829..afca6f0b4 100644 --- a/smp-web/tsconfig.test.json +++ b/smp-web/tsconfig.test.json @@ -3,7 +3,7 @@ "target": "ES2022", "module": "ES2022", "moduleResolution": "node", - "lib": ["ES2022"], + "lib": ["ES2022", "DOM"], "outDir": "dist-test", "rootDir": "tests", "strict": true, diff --git a/tests/SMPWebTests.hs b/tests/SMPWebTests.hs index 183ef1054..8b93e8805 100644 --- a/tests/SMPWebTests.hs +++ b/tests/SMPWebTests.hs @@ -46,6 +46,8 @@ import Simplex.Messaging.Encoding import Simplex.Messaging.Encoding.String (Str (..), strEncode) import Simplex.Messaging.Protocol (EntityId (..), SMPServer, SubscriptionMode (..), MsgFlags (..), noMsgFlags, pattern SMPServer, pattern NoEntity, encodeProtocol, Cmd (..), SParty (..), Command (..), NewQueueReq (..), QueueReqData (..), BrokerMsg (..), RcvMessage (..), EncRcvMsgBody (..), QueueIdsKeys (..), PubHeader (..), PrivHeader (..), ClientMessage (..), ClientMsgEnvelope (..), pattern VersionSMPC) import Simplex.Messaging.Server.Env.STM (AStoreType (..), ServerConfig (..)) +import Simplex.Messaging.Server.QueueStore.QueueInfo (QueueMode (..)) +import Simplex.Messaging.ServiceScheme (ServiceScheme (..)) import Simplex.Messaging.Server.MsgStore.Types (SMSType (..), SQSType (..)) import Simplex.Messaging.Server.Web (attachStaticAndWS) import Data.Time.Clock (getCurrentTime) @@ -1236,6 +1238,52 @@ smpWebTests_ = do <> jsOut ("e.encAgentMessage") tsResult `shouldBe` "decrypt me" + describe "agent/queueInfo" $ do + let impAgentProtoEnc = "import { encodeSMPQueueInfo, decodeSMPQueueInfo, encodeSMPQueueUri, decodeSMPQueueUri, encodeConnReqUriData, decodeConnReqUriData, encodeConnectionRequestUri, decodeConnectionRequestUri } from './dist/agent/protocol.js';" + + describe "SMPQueueInfo" $ do + it "encoding matches Haskell" $ do + g <- C.newRandom + (dhPub, _) <- atomically $ C.generateKeyPair @'C.X25519 g + let srv = SMPServer ("smp.example.com" :| []) "5223" (C.KeyHash $ B.pack [1..32]) + senderId = EntityId $ B.pack [10..33] + qAddr = AP.SMPQueueAddress {AP.smpServer = srv, AP.senderId = senderId, AP.dhPublicKey = dhPub, AP.queueMode = Just QMMessaging} + qi = AP.SMPQueueInfo (VersionSMPC 4) qAddr + hsBytes = smpEncode qi + tsBytes <- callNode $ impEnc <> impAgentProtoEnc + <> "const qi = {clientVersion: 4, queueAddress: {smpServer: {hosts: ['smp.example.com'], port: '5223', keyHash: " <> jsUint8 (B.pack [1..32]) <> "}, senderId: " <> jsUint8 (B.pack [10..33]) <> ", dhPublicKey: " <> jsUint8 (C.encodePubKey dhPub) <> ", queueMode: 'M'}};" + <> jsOut ("encodeSMPQueueInfo(qi)") + tsBytes `shouldBe` hsBytes + + it "TypeScript decodes Haskell-encoded" $ do + g <- C.newRandom + (dhPub, _) <- atomically $ C.generateKeyPair @'C.X25519 g + let srv = SMPServer ("relay.test.com" :| ["relay2.test.com"]) "" (C.KeyHash $ B.pack [50..81]) + senderId = EntityId $ B.pack [1..24] + qAddr = AP.SMPQueueAddress {AP.smpServer = srv, AP.senderId = senderId, AP.dhPublicKey = dhPub, AP.queueMode = Just QMContact} + qi = AP.SMPQueueInfo (VersionSMPC 4) qAddr + hsBytes = smpEncode qi + tsResult <- callNode $ impEnc <> impAgentProtoEnc + <> "const qi = decodeSMPQueueInfo(new Decoder(" <> jsUint8 hsBytes <> "));" + <> jsOut ("new Uint8Array([qi.clientVersion >> 8, qi.clientVersion & 0xff, qi.queueAddress.smpServer.hosts.length, qi.queueAddress.queueMode ? qi.queueAddress.queueMode.charCodeAt(0) : 0])") + tsResult `shouldBe` B.pack [0, 4, 2, 0x43] -- version=4, 2 hosts, queueMode='C' + + describe "ConnectionRequestUri" $ do + it "contact encoding matches Haskell" $ do + g <- C.newRandom + (dhPub, _) <- atomically $ C.generateKeyPair @'C.X25519 g + let srv = SMPServer ("smp1.example.com" :| []) "" (C.KeyHash $ B.pack [1..32]) + senderId = EntityId $ B.pack [1..24] + qAddr = AP.SMPQueueAddress {AP.smpServer = srv, AP.senderId = senderId, AP.dhPublicKey = dhPub, AP.queueMode = Just QMContact} + qUri = AP.SMPQueueUri (mkVersionRange (VersionSMPC 4) (VersionSMPC 4)) qAddr + crData = AP.ConnReqUriData {AP.crScheme = SSSimplex, AP.crAgentVRange = mkVersionRange (AP.VersionSMPA 2) (AP.VersionSMPA 7), AP.crSmpQueues = qUri :| [], AP.crClientData = Nothing} + cr = AP.CRContactUri crData :: AP.ConnectionRequestUri 'AP.CMContact + hsBytes = smpEncode cr + tsBytes <- callNode $ impEnc <> impAgentProtoEnc + <> "const cr = {mode: 'contact', crData: {crAgentVRange: {min: 2, max: 7}, crSmpQueues: [{clientVRange: {min: 4, max: 4}, queueAddress: {smpServer: {hosts: ['smp1.example.com'], port: '', keyHash: " <> jsUint8 (B.pack [1..32]) <> "}, senderId: " <> jsUint8 (B.pack [1..24]) <> ", dhPublicKey: " <> jsUint8 (C.encodePubKey dhPub) <> ", queueMode: 'C'}}], crClientData: null}};" + <> jsOut ("encodeConnectionRequestUri(cr)") + tsBytes `shouldBe` hsBytes + describe "protocol/e2e" $ do describe "PubHeader" $ do it "encoding without key matches Haskell" $ do