From e92f227fa77b65ecdb85da48a609d9905a8f097d Mon Sep 17 00:00:00 2001 From: spacedandy-agents-simplex <342056592+spacedandy-agents-simplex@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:16:18 +0000 Subject: [PATCH] docs: revise the several-directories plan after review rounds 1-4 --- plans/2026-10-22-directories-in-operators.md | 131 +++++++++++-------- 1 file changed, 73 insertions(+), 58 deletions(-) diff --git a/plans/2026-10-22-directories-in-operators.md b/plans/2026-10-22-directories-in-operators.md index 4b6de8c933..6de0134f8b 100644 --- a/plans/2026-10-22-directories-in-operators.md +++ b/plans/2026-10-22-directories-in-operators.md @@ -1,16 +1,16 @@ # Plan: several directories, configured like chat relays -Page for ep: (published after the review round; its link goes here) +Page for ep: https://claude.ai/artifact/YRpeaz3JcJ72onsRYy7n5G Run: 009-directory-operators · Repo: simplex-chat · Estimated diff: ~3,900 changed lines in three pull requests (core ~900, iOS ~1,550, Kotlin ~1,450, CI included), plus an optional directory pull request (~250) · Bases and order: section 6 ## Summary: what ep decides -Directories become a fifth list in each operator's servers and under Your servers, stored, preset and merged at start-up exactly like chat relays, with SimpleX Directory preset for the SimpleX Chat operator (boards 10–12). A search asks every directory in use at once and shows one list, with a retry row per failed directory and a submenu saying which directories listed a result (boards 13–17). This is plan 015's option A, re-checked against today's code, with its open review notes folded in and two answers changed: a preset its operator drops is kept turned off rather than deleted (section 4.2), and the release order (open question 3). +Directories become a fifth list in each operator's servers and under Your servers, stored, preset and merged at start-up exactly like chat relays, with SimpleX Directory preset for the SimpleX Chat operator (boards 10–12). A search asks every directory in use at once and shows one list, with a retry row per failed directory and a submenu saying which directories listed a result (boards 13–17). This is plan 015's option A, re-checked against today's code, with its open review notes folded in. Changed from plan 015: consent remembers only the operators' preset directories, a directory the user adds is agreed to by adding it, and a second alert asks before a preset not yet agreed to receives a search (section 4.4); results keep settings order, with the earliest directory's entry for a shared channel and a 15 s hold per directory; a directory is in use by its own switch, whether or not its operator's servers are on; a preset its operator drops is kept turned off rather than deleted, and that rule ships only with the first version that drops one (section 4.2); the core sends no "no directories" warning. The release order (open question 3) is a new question. -ep's newest direction, a generic MCP-style service layer in which a service fills a role when it offers the role's tools, is answered in section 4.6, against his own draft (docs/protocol/agent-protocols.md, 2026-10-19). Plainly: this design replaces "a role inferred from the tools a service offers" with "the user puts a directory in the Directories list". How a request travels (a service request) is shared, and what a request looks like (his envelope with MCP inside) can be adopted for directories whenever it is final, without moving where directories are configured. A later move to a services list keeps the storage, the search and the consent; it replaces the directory settings screens, adds a services list, migrates the rows and keeps the old settings key for a release (~1,500 lines, section 4.6.3). Telemetry, the next service ep named, has one receiver per operator and needs explicit consent, so in this design it is a switch on the operator's screen; either design needs the same privacy-policy change first (section 4.6.2). +ep's newest direction, a generic MCP-style service layer in which a service fills a role when it offers the role's tools, is answered in section 4.6, against his own draft (docs/protocol/agent-protocols.md, 2026-10-19). Plainly: this design replaces "a role inferred from the tools a service offers" with "the user puts a directory in the Directories list". How a request travels (a service request) is shared, and what a request looks like (his envelope with MCP inside) can be adopted for directories whenever it is final, without moving where directories are configured. A later move to a services list keeps the storage, the search and the consent; it replaces the directory settings screens, adds a services list, migrates the rows and keeps the old settings key for a release (~1,500 lines, section 4.6.3). Telemetry, the next service ep named, needs explicit consent; assuming each operator would receive its own, in this design it is a switch on the operator's screen, and if only the app vendor receives it, a switch in Privacy settings with one built-in address, as badges have; either design needs the same privacy-policy change first (section 4.6.2). -Decisions asked of ep (open questions 1, 2, 3, 13 and 14): this design or the service layer first; if this design, the core pull request to master right away, ideally before directory search is released; whether directory search ships before several directories (recommended) or waits for them; when the directory should answer his MCP envelope (recommended: together with the core's `protocols` declarations from his draft, not before); and whether to ship while colluding directories can link one search (recommended: yes). +Decisions asked of ep (open questions 1, 2, 3, 9, 13 and 14): this design or the service layer first; if this design, the core pull request to master right away, ideally before directory search is released; whether directory search ships before several directories (recommended) or waits for them; whether a directory follows its operator's servers switch (recommended: no, its own switch decides, so Flux-only users and self-hosters keep directory search, asked first); when the directory should answer his MCP envelope (recommended: together with the core's `protocols` declarations from his draft, not before); and whether to ship while colluding directories can link one search (recommended: yes). ## 1. Goal and acceptance criteria @@ -19,20 +19,21 @@ What the user gets, all parts together: directories are listed and added like ch Core (pull request 1; tests in the container except Postgres): - AC1: a user's servers carry a list of directories beside the chat relays. `/_servers ` returns, per operator group, `directories` with address, name, domains, preset, tested, enabled; `/_servers ` saves them (add, rename, turn off, delete). A requested name is normalised, and an empty result is stored as "" and stays "" on every later save. Test: `testGetSetDirectories` (new, tests/ChatTests/Directories.hs). -- AC2: SimpleX Directory (`https://smp4.simplex.im/a#lXUjJW5vHYQzoLYgmi8GbxkGP41_kjefFvBrdwg-0Ok`) is preset for the SimpleX Chat operator, enabled, for new users and, on the next start, for existing ones; a stored preset keeps its enabled flag across starts; a directory the user added at a preset's address becomes that preset, keeping its id and enabled flag; a stored preset that its operator, still preset, no longer lists is kept, turned off, under Your servers, where the user can delete it; a whole operator dropped from the presets, and the CLI's `-s` with `--xftp-server`, leave every stored directory as it is. Tests: tests/OperatorTests.hs cases on the start-up pass and the merge, each asserting over every group (slices 3a and 3b), and a restart test in tests/ChatTests/Directories.hs that switches configs on one database. +- AC2: SimpleX Directory (`https://smp4.simplex.im/a#lXUjJW5vHYQzoLYgmi8GbxkGP41_kjefFvBrdwg-0Ok`) is preset for the SimpleX Chat operator, enabled, for new users and, on the next start, for existing ones; a stored preset keeps its enabled flag across starts; a directory the user added at a preset's address becomes that preset, keeping its id and enabled flag; a whole operator dropped from the presets, and the CLI's `-s` with `--xftp-server`, leave every stored directory as it is. Tests: tests/OperatorTests.hs cases on the start-up pass and the merge, each asserting over every group (slices 3a and 3b), and a restart test in tests/ChatTests/Directories.hs that switches configs on one database. +- AC2b (ships with the first version that drops a preset directory, slice 3c): a stored preset that its operator, still preset, no longer lists is kept, turned off, under Your servers, where the user can delete it. Tests: slice 3c's. - AC3: two directories with the same address are an error (`USEDuplicateDirectoryAddress`), from `/_validate_servers` and on save. Tests: `validateServersTest` cases; a `/_validate_servers 1 ` step in `testGetSetDirectories`. -- AC4: compatibility both ways. A `/_servers` JSON without `directories` (an older app, or an older desktop's command through a newer phone's core) saves no change to directories and keeps the stored ones; validation can't raise a directory error without directories. A servers response without `directories` (an older phone's core, read by a newer desktop's core) parses, with no directories. Tests: a JSON test decoding `UserOperatorServers`, and `CRUserServers` through `RRResult ChatResponse`, without the key; `testGetSetDirectories` saving and validating a JSON without `directories` and reading the stored ones back. +- AC4: compatibility both ways. A `/_servers` JSON without `directories` (an older app, or an older desktop's command through a newer phone's core) saves no change to directories and keeps the stored ones; validation can't raise a directory error without directories. A servers response without `directories` (an older phone's core, read by a newer desktop's core) parses, with no directories. Tests: a JSON test decoding `UserOperatorServers`, and `CRUserServers` through `RRResult ChatResponse`, without the key; `testGetSetDirectories` saving a JSON without `directories` and reading the stored ones back (validation needs no test: the duplicate check runs over the request's directories, here none). - AC5: a new chat profile copies the current profile's directories, as it copies relays. Test: `testCreateUserSameServers` extended. - AC6: the migration goes up and down on SQLite and Postgres, the table is STRICT with its foreign key indexed, schema dumps regenerate. Tests: tests/SchemaDump.hs (SQLite, container); Postgres on the host. - AC7: no change to the bot API: no new command, response or store error, and no new warning; the generated bot API files stay the same. Test: `-m "Bot API docs"`. -- AC8: the preset lists can't break start-up: every preset directory's domains are its operator's, no address is preset twice, in `defaultChatConfig` and `terminalChatConfig`, and the SimpleX Chat operator presets the same directories in both. Test: a pure OperatorTests case over both configs. +- AC8: the preset lists can't break start-up: every preset directory's domains are its operator's, no address is preset twice, in `defaultChatConfig` and `terminalChatConfig`, and the SimpleX Chat operator presets the same directories in both; and every preset directory address shipped so far is still preset (a pinned list, so dropping one fails the test until slice 3c ships). Test: a pure OperatorTests case over both configs. Apps (pull requests 2 and 3; not compiled in the container, checked on the runners): -- AC9: settings: boards 10–12 work like chat relays on both platforms: the operator screen's "Directories" section, Your servers' section and "Directory" under Add server, the New directory screen with "Test directory" and "Use for search", and "No directory is in use" when none is. -- AC10: search: every network action (search, Load more, retry) reads the active profile's servers right before it asks for consent, and sends nothing if the profile, the remote host or the search changed meanwhile; one request per directory in use, concurrently, for that profile; a directory's rows appear once every directory before it has answered, failed or passed 15 s; for a link several directories list, the earliest directory's entry; per-directory retry; the footer rows once the load is complete; the load row across directories; at most 50 entries taken from one response; counts summed without overflow; the consent rules of section 4.4. +- AC9: settings: boards 10–12 work like chat relays on both platforms: the operator screen's "Directories" section (shown also while the operator's servers are off, checked by a device case), Your servers' section and "Directory" under Add server, the New directory screen with "Test directory" and "Use for search", and "No directory is in use" when none is. +- AC10: search: every network action (search, Load more, retry) reads the servers of the profile it captured, right before it asks for consent, and sends nothing if the profile, the remote host or the search changed by the time it sends; one request per directory in use, concurrently, for that profile; a directory's rows appear once every directory before it has answered, failed or passed 15 s (the deadline itself checked by review: a stopped test directory fails on the core's own timeout first); for a link several directories list, the earliest directory's entry; per-directory retry; Cancel as section 4.4 says; the load row across directories; at most 50 entries taken from one response; counts summed without overflow; the consent rules of section 4.4. - AC11: the long-press Directories submenu (boards 16–17). -- AC12: CI no longer edits app sources. The test backends start two directories, "CI Directory A" and "CI Directory B", with overlapping listings and one shared channel whose stored description is changed in B's database (sqlite3, while B is stopped), since a directory sends a channel's own profile text. First, with the preset still on and nothing sent: turn the switch on in Privacy settings, search, and check that the second alert names only SimpleX Directory and that Cancel leaves no spinner and no rows (proving forgetting would need agreeing, which sends to the live directory, so that is checked on a device). Then turn the preset off, add A and then B in Settings, and check: the first alert names both "(added by you)"; one list with the shared channel once, with A's description; "about N left"; with B stopped, B's own retry row and A's results kept; the submenu listing both; footer rows in both list layouts. The one-time upgrade step is checked on a device. +- AC12: CI no longer edits app sources. The test backends start two directories, "CI Directory A" and "CI Directory B", with overlapping listings and one shared channel whose `group_profiles.short_descr` is changed in B's database (sqlite3, while B is stopped, checked to survive B's restart), since a directory sends a channel's own short description. First, with the preset still on and nothing sent: turn the switch on in Privacy settings, search, and check that the second alert names only SimpleX Directory and that Cancel leaves no spinner and no rows (proving forgetting would need agreeing, which sends to the live directory, so that is checked on a device). Then turn the preset off, turn the switch off in Privacy settings, add A and then B in Settings, and check: the first alert names both "(added by you)"; one list with the shared channel once, with A's description; "about N left"; with B stopped, B's own retry row and A's results kept; the submenu listing both; footer rows in both list layouts. The one-time upgrade step is checked on a device. Directory service (optional pull request 4; container tests): @@ -74,7 +75,7 @@ Invariants to keep: presets can be turned off, never deleted by the user; a stor Directories become a fifth list of each operator group, stored and preset as chat relays are, so every layer that handles relays gains a list rather than a new concept: the operator screens get a "Directories" section where they have "Chat relays", Add server gets "Directory" where it has "Chat relay". The apps search the directories in use of the active profile instead of one constant, keeping per directory the state they keep today for one (cursor, count, failure), and merge the pages into the one list they already draw. -**In use:** a directory is in use when it is enabled, not deleted, and its group's operator is enabled (the custom group counts as enabled). One rule decides who receives a search, who the alerts name, whether "Search in Directory" shows, and the "No directory is in use" note. A user who turns the SimpleX Chat operator off sends no searches to SimpleX Directory, as channel creation already skips a disabled operator's relays (open question 9). The apps compute the rule from `/_servers`. +**In use:** a directory is in use when it is enabled ("Use for search") and not deleted, whether or not its operator's servers are turned on. One rule decides who receives a search, who the alerts name, whether "Search in Directory" shows, and the "No directory is in use" note. An operator's "Use servers" switch is about its servers for messages; a directory is a service the user agrees to by name in the consent alert, so a user who chose only Flux at onboarding, or runs their own servers with both operators off, still finds SimpleX Directory, asked first (open question 9). The apps compute the rule from `/_servers`. ### 4.2 Core: storage, presets, start-up @@ -83,17 +84,17 @@ New mechanisms, each with its requirement: - Type `UserDirectory' s {directoryId :: DBEntityId' s, address :: ShortLinkContact, displayName :: Text, domains :: [Text], preset :: Bool, tested :: Maybe Bool, enabled :: Bool, deleted :: Bool}`, with `UserDirectory`, `NewUserDirectory`, `AUserDirectory = AUD (SDBStored s) (UserDirectory' s)` — boards 10–12 (a list with name, test state and "Use for search"). `deleted` is the API's delete-on-save flag, as for relays; there is no `deleted` column (nothing references a directory, so delete is a plain DELETE). - `directories :: ListDef UserDirectory` in `UserOperatorServers` and `directories :: ListDef AUserDirectory` in `UpdatedUserOperatorServers`, so a servers response or request without the key (an older phone's core in a remote session; an older app saving) parses as an empty list. An empty list in a request saves nothing and keeps the stored rows (a save only upserts the listed rows and deletes the ones flagged `deleted`) and can raise no directory error, so no "app doesn't know directories" state is needed. `ADirectory`, `directories'`, `aUserDirectory'` in `UserServersClass`. Rejected: a plain list (a missing key fails to parse); `Maybe` in the request (it would distinguish nothing); a hand-written parser with `.:? … .!= []` (an instance where the codebase already has a type). - Table `directories (directory_id INTEGER PRIMARY KEY, address BLOB NOT NULL, display_name TEXT NOT NULL, domains TEXT NOT NULL, preset INTEGER NOT NULL DEFAULT 0, tested INTEGER, enabled INTEGER NOT NULL DEFAULT 1, user_id INTEGER NOT NULL REFERENCES users ON DELETE CASCADE, created_at, updated_at) STRICT`, `idx_directories_user_id`, unique `idx_directories_user_id_address`; Postgres with `BIGINT … GENERATED ALWAYS AS IDENTITY`, `BYTEA`, `SMALLINT` as `chat_relays` — per-profile storage, "stored like chat relays" (the teal note). The cascade removes a deleted profile's rows. `domains` is comma-joined text as for relays; the store reads "" as `[]`. `getDirectories` orders by `directory_id`, and grouping takes the list reversed, as it does for servers (it prepends each row), so every group keeps the order in which its directories were added; that is the "settings order" of section 4.4. The id follows `chat_relays`; master switched eight entity tables to AUTOINCREMENT on 2026-10-19 without a stated reason, so the core pull request's reviewer should confirm (nothing keeps a directory's id beyond one save: the apps identify directories by address). -- Presets: `simplexDirectories :: [NewUserDirectory]` in Presets.hs, each built from its operator (`presetDirectory operatorSimpleXChat "SimpleX Directory" ` takes the operator's `serverDomains` as its domains, so a preset can't name another operator); a `directories` field in `PresetOperator`, set for the SimpleX Chat operator in both `defaultChatConfig` and `terminalChatConfig` (the same list: a config that presets fewer directories under the same operator would retire the others on each start); Flux and the CLI's custom preset have none — "SimpleX Directory preset" (the orange note). A pure test asserts, for both configs, that every preset directory's domains are its operator's and that no address is preset twice (AC8): either mistake would make the operator's merge insert a row that already exists and fail every start. +- Presets: `simplexDirectories :: [NewUserDirectory]` in Presets.hs, each built from its operator (`presetDirectory operatorSimpleXChat "SimpleX Directory" ` takes the operator's `serverDomains` as its domains, so a preset can't name another operator); a `directories` field in `PresetOperator`, set for the SimpleX Chat operator in both `defaultChatConfig` and `terminalChatConfig` (the same list: once the retire rule ships, a config that presets fewer directories under the same operator would retire the others on each start); Flux and the CLI's custom preset have none — "SimpleX Directory preset" (the orange note). A pure test asserts, for both configs, that every preset directory's domains are its operator's and that no address is preset twice (AC8): either mistake would make the operator's merge insert a row that already exists and fail every start. The same test pins the list of preset directory addresses shipped so far and fails when one disappears, with a message pointing to slice 3c: a version that drops a preset without the retire rule would leave it stored, enabled and remembered, still receiving searches. - One pure pass over the user's flat list of stored directories at start-up, before grouping, in `getServers` (Chat.hs:309-318), with two rules, in this order: - 1. **Adopt.** A row at the address of a directory preset by an operator with a preset entry in the start-up list (`sameShortLinkContact`) is rewritten to that preset's `domains` with `preset = True`, keeping its id, name, tested and enabled flag. Grouping by domains then puts it in the operator's group only, and the per-group merge stays exactly like relays' (a preset matched within its group). This is what makes a third-party directory becoming a preset safe: without it the preset would be inserted beside the user's row and hit the unique index inside the start-up transaction. If that operator is turned off, the adopted directory stops being searched and its section is hidden until the operator is turned on (the in-use rule; open question 9). Rejected: adopting inside `updatedUserServers` (the row stays in the custom group's result too, and the custom group, saved last, turns it back on every start); an upsert on insert (the same save-order problem). - 2. **Retire.** A row with `preset = True` whose domains name an operator with a preset entry in the start-up list, and whose address that operator no longer presets, is rewritten to `preset = False`, `domains = []`, `enabled = False`, keeping its id and name. Grouping then puts it under Your servers, where it stays turned off and the user can delete it (the operator screen has no delete). An operator that drops a directory thereby stops the searches it gets, with no new store function; if a later version presets the address again, rule 1 adopts it, still off. Scoping by operators with a preset entry keeps `-s` with `--xftp-server` (no preset operator at all), the CLI and bots (which don't preset Flux), and untagged operators the user made from touching any stored row. Rejected: deleting the row (plan 015's answer): a store delete outside the preset guard, and a later re-preset would insert the address enabled while the consent set (section 4.4) already holds it, so a directory the user had turned off would receive searches without a question. + 1. **Adopt.** A row at the address of a directory preset by an operator with a preset entry in the start-up list (`sameShortLinkContact`) is rewritten to that preset's `domains` with `preset = True`, keeping its id, name, tested and enabled flag. Grouping by domains then puts it in the operator's group only, and the per-group merge stays exactly like relays' (a preset matched within its group). This is what makes a third-party directory becoming a preset safe: without it the preset would be inserted beside the user's row and hit the unique index inside the start-up transaction. The adopted directory moves from Your servers to that operator's section, keeping its switch. Rejected: adopting inside `updatedUserServers` (the row stays in the custom group's result too, and the custom group, saved last, turns it back on every start); an upsert on insert (the same save-order problem). + 2. **Retire** (ships with the first version that drops a preset directory: open question 12). A row with `preset = True` whose domains name an operator with a preset entry in the start-up list, and whose address that operator no longer presets, is rewritten to `preset = False`, `domains = []`, `enabled = False`, keeping its id and name. Grouping then puts it under Your servers, where it stays turned off and the user can delete it (the operator screen has no delete). An operator that drops a directory thereby stops the searches it gets, with no new store function; if a later version presets the address again, rule 1 adopts it, still off. Scoping by operators with a preset entry keeps `-s` with `--xftp-server` (no preset operator at all), the CLI and bots (which don't preset Flux), and untagged operators the user made from touching any stored row. Rejected: deleting the row (plan 015's answer): a store delete outside the preset guard, and a later re-preset would insert the address enabled while the consent set (section 4.4) already holds it, so a directory the user had turned off would receive searches without a question. Invariants: the pass never changes a row with `preset = False` except to adopt it; it never retires a row at any preset's address; apart from adopting a row at a preset's address, it leaves every row alone whose domains name no operator with a preset entry. Directories of an operator a later version drops entirely stay stored and in use, as its servers and relays do today (Operators.hs:464-465); the plan claims nothing more. - Costs of retiring, accepted: the pass can't tell "the operator dropped it" from "this binary presets less". An older binary that already has the table (a later version's CLI is newer than an app, or the reverse) opening the same database retires the directories it doesn't preset, and the next start of the newer one adopts them back turned off, though the user never turned them off. A directory the user added, adopted as a preset by one version and dropped by a later one, comes back turned off. Both are silent and fail safe (off, so no search text goes anywhere new). + Costs of retiring, accepted when it ships: the pass can't tell "the operator dropped it" from "this binary presets less". A binary with the retire rule that presets fewer directories (an older app or CLI opening a newer database) retires the directories it doesn't preset, and the next start of the newer one adopts them back turned off, though the user never turned them off. A directory the user added, adopted as a preset by one version and dropped by a later one, comes back turned off, still labelled "(added by you)" if the user turns it on again. Both are silent and fail safe (off, so no search text goes anywhere new). Shipping retire only with the first version that drops a preset keeps this cost away until then; adopt has no such cost (it only rewrites rows at its own presets' addresses) and guards a start-up failure, so it ships now. - Merge in `updatedUserServers`: `updatedDirectories`, every preset directory of the group's operator matched by address to the group's stored rows (kept as stored, so a renamed preset keeps its stored name, as relays do) or new, plus the group's stored custom rows; the same shape as `updatedRelays` (Operators.hs:434-447). -- Validation: `USEDuplicateDirectoryAddress {duplicateDirectory :: Text, duplicateAddress :: ShortLinkContact}` over the request's groups — so the settings screens can mark a duplicate. A request without directories carries an empty list, so it can't raise this error; no other user's directories are checked (other users get only missing-server errors today). No "no directories" warning: an app shows that note from its own data (section 4.3), which needs no new case in the apps' closed warning types and doesn't nag a profile whose user turned directories off on purpose. -- A directory's name is normalised where app input enters the core: the `APISetUserServers` and `APIValidateServers` handlers pass each requested directory's name through `mkValidName` (Commands.hs:6948 / 6968) before validating and saving; an empty result is stored as "", and the apps show "Unnamed directory" where the name is empty (not the address's host, which is its router and can look official, section 4.7). The name comes from the directory's own profile and is shown in the consent alert, so this bounds what a directory can put there (section 5.4). +- Validation: `USEDuplicateDirectoryAddress {duplicateDirectory :: Text, duplicateAddress :: ShortLinkContact}` over the request's groups — so the settings screens can mark a duplicate. A request without directories carries an empty list, so it can't raise this error; no other user's directories are checked (other users get only missing-server errors today). No "no directories" warning: an app shows that note from its own data (section 4.3), which needs no new case in the apps' closed warning types and no "For chat profile X:" warnings about other profiles. +- A directory's name is normalised where app input is stored: the `APISetUserServers` handler passes each requested directory's name through `mkValidName` (Commands.hs:6948 / 6968) before saving; an empty result is stored as "", and the apps show "Unnamed directory" where the name is empty (not the address's host, which is its router and can look official, section 4.7). The name comes from the directory's own profile and is shown in the consent alert, so this bounds what a directory can put there (section 5.4). - `copyServers` (Commands.hs:501-511) copies directories to a new profile as new rows, as it copies relays. - The CLI view gains a "Directories" part in `viewUserServers` (View.hs:1668-1674). @@ -105,27 +106,27 @@ No new command, response, store error or warning: the apps learn a directory's n As drawn, built from `ChatRelayView`'s shared pieces, not a copy of it: -- Board 10: the operator screen's "Directories" section below "Chat relays", shown when the operator is enabled and has directories; footer "Directories answer your searches for public groups and channels."; rows open `DirectoryView` (a preset: read-only address and name, "Use for search"). +- Board 10: the operator screen's "Directories" section below "Chat relays", shown whenever the operator has directories, also while its servers are turned off (unlike the relays section, since a directory is in use by its own switch); footer "Directories answer your searches for public groups and channels."; rows open `DirectoryView` (a preset: read-only address and name, "Use for search"). - Board 11: Your servers gets a "Directories" section with delete, and "Directory" after "Chat relay" in Add server. - Board 12: New directory: "Your directory address", "Your directory name" (read-only, filled by the test), "Test directory" with its result, "Use for search". Added below the address: "The directory receives the text of each search, without your profile." — what a user is told about a directory they add, and the consent for it (section 4.4). -- "Test directory" runs two steps in the apps: the address's connect plan (`APIConnectPlan`; it must be a contact address; its profile's display name fills the name) and one search request with empty text (it must answer with search results; empty text matches every listing, so no words are sent). A failure says which step failed. No core command, so the core stays unaware of the directory's JSON. "Test servers" on the operator and Your servers screens tests enabled directories the same way, whatever the Directory search switch says: it sends no words, only the empty search the user asked for by pressing it. +- "Test directory" runs two steps in the apps: the address's connect plan (`APIConnectPlan`; it must be a contact address; its profile's display name fills the name) and one search request with empty text (it must answer with search results; empty text matches every listing, so no words are sent). A failure says which step failed. No core command, so the core stays unaware of the directory's JSON. "Test servers" on the operator and Your servers screens tests enabled directories the same way, whatever the Directory search switch says: it sends no words, only the empty search the user asked for by pressing it. On an operator screen it stays inside the operator-enabled part, as today, so while an operator's servers are off its directories are tested from their own screens. - When no directory is in use, Network & servers says "No directory is in use, so search finds only your chats." beside its other notes, computed in the app. ### 4.4 Search with several directories (boards 13–17) -- **Which directories.** Every network action (a search, Load more, a retry) reads the active profile's servers (`/_servers`, a local call) right before it asks for consent and sends, and computes from them the directories it will contact: a search, every directory in use; Load more, the search's directories still in use that have more; a retry, that directory if still in use. It captures, with that read, the user id, the remote host and the model's generation; after any alert it sends nothing if one of them changed (a profile switch from a notification tap, a reset), and every request it sends carries that user id rather than looking up the current user again. "Search in Directory" and the Search key use a model value refreshed when chat starts, the active user or remote host changes, servers are saved (Network & servers and the submenu's sheet) and operators change; a stale value can only show or hide the row, never decide where text goes. If the fresh read finds no directory in use, nothing is sent. +- **Which directories.** Every network action (a search, Load more, a retry) first captures the active user id, the remote host and the model's generation, then reads that user's servers (`/_servers `, a local call) and computes from them the directories it will contact: a search, every directory in use; Load more, the search's directories still in use that have more; a retry, that directory if still in use. Right before sending, after any alert or none, it checks that the user, the remote host and the generation are still the captured ones and sends nothing otherwise (a profile switch from a notification tap, a reset); every request carries the captured user id rather than looking up the current user again. "Search in Directory" and the Search key use a model value refreshed when chat starts, the active user or remote host changes, and servers are saved (inside `saveServers`, which every save path uses: the Save button, the "Save servers?" alert on closing, the submenu's sheet); a stale value can only show or hide the row, never decide where text goes. If the fresh read finds no directory in use, nothing is sent. - **Sending.** One request per directory, all at once, each with today's 10 s timeout (which the core starts after fetching the address's link data, so a request to an unreachable server takes longer). iOS sends each blocking call from a dispatch queue rather than the cooperative pool (otherwise a few slow directories stall the app's async work for that long); Kotlin already sends on `Dispatchers.IO`. The sender knows nothing of directories (one request to many addresses, each answer or failure reported), so another role can reuse it (section 4.6.3). - **State.** `OnlineSearchModel` keeps, per directory address: name, whether the user added it, cursor, count, loading, failed. The results reset on a profile switch in the chat list view, unchanged. - **Order.** Your chats first (as built); then each load's results in settings order (the operators' directories in operator order, then yours, each group in the order its directories were added), each in the directory's own order. A directory's rows appear once every directory before it in that order has answered or failed in this load, so results already shown don't move. The app gives each directory 15 s from the start of the load (its 10 s plus time to fetch the link): one that hasn't answered by then counts as failed, gets its retry row, and its late answer is ignored; so a slow earlier directory delays the later ones' rows by at most 15 s. A link several directories list shows once, with the entry of the earliest directory in settings order: within a load the hold rule gives that for free; when an earlier directory lists, in a later load, a link already shown from a later one, the row keeps its place and takes the earlier directory's name, description, image and counts (Open goes to the same link). The result remembers every directory that listed it (for the submenu). One exception, as built: a result that is one of your chats joins your chats at the top. No sorting across directories (open question 6). -- **Cancel.** The spinner's Cancel (today it drops the running request) marks every directory that hasn't answered as failed, each with its retry row, and shows the held rows of those that have; the same on both platforms. -- **Footer.** The footer rows appear when the load is complete (every directory answered, failed or past its 15 s), so they don't move as rows are added above them. One retry row per failed directory, in settings order: with one directory in use, today's "Search failed, tap to retry"; with two or more, "[name]: search failed, tap to retry", which retries that directory's last request alone while the others' results stay (board 15). Then the load row, last, when any directory that hasn't failed has more: "Load more · N left" when one directory sent a count, "Load more · about N left" when two or more did (N their sum, saturated, summed in 64 bits on both platforms; "about" because a group listed by two is counted twice), "Load more" without counts (open question 5). The spinner in the field shows while any request runs (after 1 s, as built); the load row greys out while a load runs. +- **Cancel.** The spinner's Cancel: with one directory in use, as today (the request's answer is ignored; nothing new shows; a cancelled first search brings "Search in Directory" back, a cancelled Load more keeps its load row). With several, it shows the held rows of the directories that have answered, and gives each that hasn't its own row "[name]: no answer, tap to retry"; the same on both platforms. +- **Footer.** The retry rows of the first load appear when that load is complete (every directory answered, failed or past its 15 s), so they don't move as rows are added above them; during a Load more, earlier retry rows hide and the load row stays, greyed, as the designer drew, moving down as released rows are added above it (it can't be tapped while greyed). The spinner in the field follows the load, from its start until it completes or is cancelled. One retry row per failed directory, in settings order: with one directory in use, today's "Search failed, tap to retry"; with two or more, "[name]: search failed, tap to retry", which retries that directory's last request alone while the others' results stay (board 15). Then the load row, last, when any directory that hasn't failed has more: "Load more · N left" when one directory sent a count, "Load more · about N left" when two or more did (N their sum, saturated, summed in 64 bits on both platforms; "about" because a group listed by two is counted twice), "Load more" without counts (open question 5). A tap loads the next page from every such directory. The spinner appears after 1 s, as built. - **Bounds.** At most 50 entries are taken from one response (the directory sends 10); the rest are dropped. Each kept entry costs a local connection plan, so this caps the local work one directory can cause per page. - **Consent** (board 13). The one "Directory search" switch in Privacy settings stays, device-wide. Beside it the apps remember the addresses of the operators' preset directories the user has agreed to: public addresses, the same for every user, never a directory a user added. A directory the user adds is agreed to by adding it or turning it on, on the screen that says what it receives. Before an action sends: - with the switch off: "Enable Directory search?" — "Search text is sent to SimpleX Directory and Your directory (added by you), each search separately and without your profile." — naming every directory the action contacts, the user's own marked "(added by you)"; Enable turns the switch on, remembers the preset ones and sends; Cancel sends nothing; - - with the switch on, if the action would contact a preset directory not remembered (an app update presets one, the user turns an operator on, or the switch was turned on in Privacy settings, which names nothing): "Search in these directories?" — "Search text is sent to [names], each search separately and without your profile." — naming only those; Search remembers them and sends to every directory of the action; Cancel sends nothing and leaves the results as they are; - - turning the switch off forgets the remembered addresses. + - with the switch on, if the action would contact a preset directory not remembered (an app update presets one, the user turns a preset back on, or the switch was turned on in Privacy settings, which names nothing): "Search in these directories?" — "Search text is sent to [names], each search separately and without your profile." — naming only those; Search remembers them and sends to every directory of the action; Cancel sends nothing and leaves the results as they are; + - turning the switch off forgets the remembered addresses, and every servers save (in `saveServers`, so every save path) forgets any remembered address that is not an enabled preset of the profile just saved, so a preset the user turns off, or a retired one the user deletes and a later version presets again, is asked about before its next search. Accepted: the set is device-wide and directories are per profile, so a user who keeps a preset on in one profile and off in another is asked again in the first after saving the second's servers. - Board 13's last clause, "so they can't build your search history", is dropped (it no longer holds: section 5.4). With one directory in use, the first alert is today's wording with that directory's name, marked "(added by you)" when it is the user's. If directory search ships a release before several directories (open question 3, recommended), a one-time step on the first start of the new version remembers exactly the address that release searched (today's `DIRECTORY_SERVICE_LINK`, kept as a literal in the step), only if the switch is on (the user agreed to SimpleX Directory by name), and records in its own key that it ran, which turning the switch off doesn't clear, so the step never runs again. Accepted: a down migration drops the table while the remembered address stays, so after a downgrade and an upgrade SimpleX Directory comes back enabled and receives searches without a question, though the user may have turned it off before; a down migration loses every directory setting anyway, and the user agreed to that directory by name. + Board 13's last clause, "so they can't build your search history", is dropped (it no longer holds: section 5.4). With one directory in use, the first alert is today's wording with that directory's name, marked "(added by you)" when it is the user's. If directory search ships a release before several directories (open question 3, recommended), a one-time step on the first start of the new version remembers exactly the address that release searched (today's `DIRECTORY_SERVICE_LINK`, kept as a literal in the step), only if the switch is on (that release already sent their searches to SimpleX Directory, whether they turned search on from the alert or in Privacy settings), and records in its own key that it ran, which turning the switch off doesn't clear, so the step never runs again. Accepted: a down migration drops the table while the remembered address stays, so after a downgrade and an upgrade SimpleX Directory comes back enabled and receives searches without a question, though the user may have turned it off before; a down migration loses every directory setting anyway, and that directory had received the user's searches before. - **Submenu** (boards 16–17). With two or more directories in use, the long-press menu gains "Directories ›" listing the directories that listed the result, the user's own marked "(added by you)"; a tap opens that directory's settings screen in a sheet, which saves like Network & servers. With one directory in use the menu keeps Open only (as built). It is also how a user sees where a result came from. - **Remote desktop.** The desktop reads the phone's profile's directories through `/_servers` and sends each request through the phone's core, as today. Its consent switch and remembered addresses are the desktop's own. @@ -143,7 +144,7 @@ ep's direction makes a role a set of required tools and lets any service whose t #### 4.6.2 Telemetry in this design -- Configured: each operator that wants it presets one telemetry address in its `PresetOperator` (not user-added, so no list); the user turns it on per operator on the operator's screen ("Send usage statistics", off by default), stored as one more column of `server_operators`, which is device-wide like the operators themselves. A sketch of that screen is on the page. +- Configured: ep named telemetry but not its receiver. Assuming each operator that wants it receives its own: it presets one telemetry address in its `PresetOperator` (not user-added, so no list); the user turns it on per operator on the operator's screen ("Send usage statistics", off by default), stored as one more column of `server_operators`, which is device-wide like the operators themselves. A sketch of that screen is on the page. If only the app vendor receives it, it sits where badges do (section 4.6.1): one address in the core config and one switch in Privacy settings, at about the same cost. - Run: a core worker, like the badge retry worker, sends an unsigned service request to each consenting operator's address at most once a day at a random time, with aggregates only (app version and platform, feature switches, counts in coarse buckets), no identifiers, each time on a new connection. - What the operator can learn or link: the request carries no profile, but its reply queue is on one of the user's servers (a self-hosting user names the same host every time), and a rare combination of buckets can single out a device; a request needs one profile's servers, so the worker must use the same profile every time or the operator could correlate profiles. These hold in any design. - Before any of this: the privacy policy, which the core shows as the operators' usage conditions, says the apps "should not include any tracking or analytics code" and "must not send anything not directly required for users communications without explicit users' consent" (PRIVACY.md:201, :203). Telemetry needs that text changed, a new conditions version users review, and explicit consent, whichever design carries it. @@ -200,7 +201,7 @@ One migration, `M2026MMDD_directories`, dated after the newest migration on mast - An older app (or an older desktop's command through a newer phone's core) sends `/_servers ` and `/_validate_servers` without `directories`: the core parses an empty list, saves no change to directories, and returns no directory error (AC4). - An older phone's core answers a newer desktop's core: no `directories` key; `ListDef []` parses (AC4's JSON test). The desktop shows no directories for that phone and can't search through it, until the phone updates. If the core pull request lands before directory search is released, that release's phones already answer with directories and SimpleX Directory preset, so this case can't arise with any phone that has directory search. - A newer core answers an older app or an older desktop's core: the key is ignored. A newer desktop saving through an older phone loses directories silently (the phone ignores the key); harmless. -- An older desktop of a search-only release, connected to a newer phone, still sends every search to the address built into it, through the phone's core, which can't tell a directory search from any service request: SimpleX Directory receives that desktop's searches even if the phone's user turned it or its operator off. The desktop's own consent still applies. Until the desktop updates. Exists only if directory search ships before several directories (open question 3). +- An older desktop of a search-only release, connected to a newer phone, still sends every search to the address built into it, through the phone's core, which can't tell a directory search from any service request: SimpleX Directory receives that desktop's searches even if the phone's user turned it off. The desktop's own consent still applies. Until the desktop updates. Exists only if directory search ships before several directories (open question 3). - The apps decode `directories` with their normal decoding (Swift synthesized; Kotlin default `emptyList()`): an app only ever reads its own core, which always sends the key (a desktop's core re-encodes a phone's reply with it). They add `duplicateDirectoryAddress` to their closed error types in the same release that the core can send it. ### 5.4 Threat model @@ -211,7 +212,7 @@ Who can do what, and what bounds it. Today's only directory is the SimpleX Chat |---|---|---|---| | A directory the user added | Receives every search text, unsigned, without a profile, with a reply queue on one of the user's servers; may store the texts (the SimpleX Directory policy says it stores search requests, PRIVACY.md:126) | The New directory screen says what it receives; it is named "(added by you)" in the first alert; turning it off stops it at the next action (each action re-reads the servers) | runner: alert names both test directories; settings off → no request | | A hostile reply | A large, deep or malformed body; hundreds of entries; a huge count; bad links; any image up to the body bound (decoded with no dimension check, the same exposure as any profile image today) | The core: ≤ 10,737 bytes compressed, ≤ 64 KiB decompressed, depth ≤ 32; the apps: entries decoded one by one, bad ones dropped, ≤ 50 taken, counts clamped and summed without overflow, only group and channel links | the core's parser tests; the apps' decoding, cap and sums checked by review (the apps have no unit tests for this, by the team's choice) | -| A slow directory | Holds a request; under the hold rule delays later directories' rows | At most 15 s from the start of the load, then its own retry row | runner: one test directory stopped | +| A slow directory | Holds a request; under the hold rule delays later directories' rows | At most 15 s from the start of the load, then its own retry row | runner: one test directory stopped (it fails on the core's own timeout); the 15 s deadline by review | | A fast directory | Answers first to put its claims (name, description, image, counts) on channels another directory lists | The earliest directory in settings order wins for a shared link; the operators' directories come first | runner: a shared channel described differently by the two test directories | | A look-alike | A user-added directory calls itself "SimpleX Directory"; it lists a channel named like a known one | Names normalised by `mkValidName` (≤ 50 characters, no direction-changing or format characters); "(added by you)" in the alert and the submenu; the submenu shows who listed a result | `testGetSetDirectories` (normalised name) | | The directory's own server | Sees the user's IP when it fetches the address's link data or receives the request, if private routing doesn't cover that server | The user's private-routing setting: by default the apps always reach other servers through a proxy | none (documented) | @@ -223,7 +224,7 @@ Who can do what, and what bounds it. Today's only directory is the SimpleX Chat Pull requests and their order: -1. **Core** (slices 1–3b), base master, once open question 1 is this design, and ideally before directory search is released (section 5.3). It changes nothing users see. +1. **Core** (slices 1–3b; 3c later, with the first version that drops a preset), base master, once open question 1 is this design, and ideally before directory search is released (section 5.3). It changes nothing users see. 2. **iOS** (slices 5–8), base master after directory search (#7588) and the core pull request have merged (or the search branch with master merged in after the core lands). 3. **Kotlin** (slices 9–12), the same base, after iOS (iOS first, per the team's rule). 4. **Directory MCP** (slice 4), optional, base the search branch or master after #7588 (the directory's service-request code exists only there), any time after. @@ -246,19 +247,19 @@ The smallest useful first release is pull requests 1, 2 and 3: without the setti - Size and risk: medium · high (migration: database format) - Estimate: ~90 min -- Files: src/Simplex/Chat/Store/SQLite/Migrations/M2026MMDD_directories.hs and its Postgres twin (new), both Migrations.hs, simplex-chat.cabal, src/Simplex/Chat/Store/Profiles.hs (`getDirectories` ordered by `directory_id`, `insertDirectory`, `updateDirectory`, `upsertOrDeleteDirectory`; `getUserServers` reads directories; `setUserServers'` saves the listed ones), src/Simplex/Chat/Operators.hs (grouping directories by `domains` in `groupByOperator_`, as `addChatRelay` does at :516-521, over the list reversed as servers are at :502), src/Simplex/Chat/Library/Commands.hs (name normalisation in `APISetUserServers` and `APIValidateServers`), src/Simplex/Chat/View.hs (a "Directories" part), tests/ChatTests/Directories.hs (new, registered as `describe "directory tests"` beside "chat relay tests" in tests/ChatTests.hs, and in simplex-chat.cabal). +- Files: src/Simplex/Chat/Store/SQLite/Migrations/M2026MMDD_directories.hs and its Postgres twin (new), both Migrations.hs, simplex-chat.cabal, src/Simplex/Chat/Store/Profiles.hs (`getDirectories` ordered by `directory_id`, `insertDirectory`, `updateDirectory`, `upsertOrDeleteDirectory`; `getUserServers` reads directories; `setUserServers'` saves the listed ones), src/Simplex/Chat/Operators.hs (grouping directories by `domains` in `groupByOperator_`, as `addChatRelay` does at :516-521, over the list reversed as servers are at :502), src/Simplex/Chat/Library/Commands.hs (name normalisation in `APISetUserServers`), src/Simplex/Chat/View.hs (a "Directories" part), tests/ChatTests/Directories.hs (new, registered as `describe "directory tests"` beside "chat relay tests" in tests/ChatTests.hs, and in simplex-chat.cabal). - Change: the table of section 4.2; a new entry inserts, a stored one updates, a stored one flagged `deleted` is deleted when not a preset (`… AND preset = 0`, as relays); an empty list saves nothing; requested names normalised, empty kept "". - Tests: schema dump, down migrations, STRICT and foreign-key-index checks regenerate (tests/SchemaDump.hs; the dumps are regenerated by the tests, not edited); `testGetSetDirectories`, asserting values read with `APIGetUserServers` (as `enableNamesRole` does) rather than view lines: add a custom directory through `/_servers 1 `, rename it, turn it off, delete it, reading back each time; a name with prohibited characters is stored normalised; a name that normalises to nothing is stored "" and still "" after a second save; two directories added in turn read back in that order after another save; a JSON without `directories` keeps them. - Verify: build, then `-m "Schema dump"`, `-m "directory tests"`, plus slice 1's commands. - Done when: the library and tests build, the schema tests pass with the regenerated SQLite dump, and AC1 and AC4's save case pass. -#### Slice 3a: the start-up pass (behavioural) +#### Slice 3a: the start-up pass with the adopt rule (behavioural) - Size and risk: small · high (it runs for every user on every start; inert here, as nothing presets a directory yet) -- Estimate: ~60 min +- Estimate: ~45 min - Files: src/Simplex/Chat/Operators.hs (the pass, pure), src/Simplex/Chat.hs (`getServers` reads directories, runs the pass, then groups), tests/OperatorTests.hs. -- Change: section 4.2's two rules, wired before grouping. -- Tests (pure, with test presets): adopt (a custom row, `domains = []`, `enabled = False`, at a preset's address comes back with the operator's domains, `preset = True`, its id and `enabled = False`); adopt under a turned-off operator (same rewrite; the row lands in that operator's group); retire (a preset its operator no longer lists comes back `preset = False`, `domains = []`, `enabled = False`, its id); `-s` with `--xftp-server` (the start-up list holds only the custom preset) leaves every row unchanged; an operator missing from the presets (a Flux row under the CLI's config) leaves its rows unchanged; an untagged operator without a preset entry leaves its rows unchanged; a turned-off preset whose address another operator now presets is adopted by that operator, still off, and not retired; a QuickCheck property over generated rows and presets: never a changed row with `preset = False` other than an adoption, never a retired row at any preset's address. +- Change: section 4.2's adopt rule, wired before grouping; the retire rule is slice 3c. +- Tests (pure, with test presets): adopt (a custom row, `domains = []`, `enabled = False`, at a preset's address comes back with the operator's domains, `preset = True`, its id and `enabled = False`); adopt under a turned-off operator (same rewrite; the row lands in that operator's group and stays enabled if it was); `-s` with `--xftp-server` (the start-up list holds only the custom preset) leaves every row unchanged; an operator missing from the presets (a Flux row under the CLI's config) leaves its rows unchanged; an untagged operator without a preset entry leaves its rows unchanged; a preset whose address another operator now presets is adopted by that operator with its enabled flag; a QuickCheck property over generated rows and presets: a row changes only if it is at a preset's address, and then only its domains and preset flag. - Verify: build `simplex-chat-test`, then `-m "Operators"`, plus slice 1's commands. - Done when: all pass; one lean review of slices 1–3a before 3b builds on them. @@ -269,13 +270,21 @@ The smallest useful first release is pull requests 1, 2 and 3: without the setti - Files: src/Simplex/Chat/Operators.hs (`presetDirectory` from the operator, `newDirectory`, `updatedDirectories`, `presetUserServers` (:391) taking the operator's directories, `USEDuplicateDirectoryAddress` with slice 1's fold), src/Simplex/Chat/Operators/Presets.hs (`simplexDirectories`), src/Simplex/Chat.hs (the SimpleX Chat operator's `directories = simplexDirectories` in `defaultChatConfig`), src/Simplex/Chat/Terminal.hs (the same), src/Simplex/Chat/Library/Commands.hs (`copyServers` copies directories). - Change: section 4.2's presets, merge and validation. - Tests: - - tests/OperatorTests.hs, the merge over every group: the preset added to stored groups without directories; a stored preset with `enabled = False` stays off; an adopted row appears exactly once, in the operator's group, and the same again on a second merge of the saved result; a retired row appears once, in the custom group, and stays so on a second pass; the presets' invariants over `defaultChatConfig` and `terminalChatConfig` (AC8). - - `validateServersTest`: duplicate address error; duplicate names allowed; a request whose groups carry no directories raises no directory error. Changed expectations: none (no new warning). - - tests/ChatTests/Directories.hs: `testGetSetDirectories` gains `/_validate_servers 1 ` steps (a duplicate address gives the error; a JSON without `directories` gives none, with a second user present); a restart test on one database: start with a config whose SimpleX Chat operator presets a test directory, `smpServers = [], xftpServers = []` (presets in use offline, as `testOperators` does, Direct.hs:1226-1254), add a custom directory; restart with `-s` and `--xftp-server`: both unchanged; restart with a config whose operator presets none: the preset is under Your servers, turned off, the custom one unchanged; restart with the first config: the preset is back under the operator, still off. Values are read with `APIGetUserServers`, not view lines (real presets print random "disabled" markers). + - tests/OperatorTests.hs, the merge over every group: the preset added to stored groups without directories; a stored preset with `enabled = False` stays off; an adopted row appears exactly once, in the operator's group, and the same again on a second merge of the saved result; the presets' invariants over `defaultChatConfig` and `terminalChatConfig` (AC8). + - `validateServersTest`: duplicate address error; duplicate names allowed. Changed expectations: none (no new warning). + - tests/ChatTests/Directories.hs: `testGetSetDirectories` gains a `/_validate_servers 1 ` step (a duplicate address gives the error); a restart test on one database: start with a config whose SimpleX Chat operator presets a test directory, `smpServers = [], xftpServers = []` (presets in use offline, as `testOperators` does, Direct.hs:1226-1254), add a custom directory, then a second custom one at an address a second config presets; restart with `-s` and `--xftp-server`: all unchanged; restart with the second config: the second custom directory is that preset now, under the operator, with its switch as it was. Values are read with `APIGetUserServers`, not view lines (real presets print random "disabled" markers). - tests/ChatTests/Direct.hs: `testCreateUserSameServers` checks a second profile gets the first one's directories; `testOperators` checks `/_servers 1` lists SimpleX Directory under SimpleX Chat (by value). - Verify: build `simplex-chat-test`, then `-m "Operators"`, `-m "directory tests"`, `-m "get and enable operators"`, `-m "chat relay tests"`, `-m "get and set SMP servers"`, `-m "create user with same servers"`, `-m "JSON Tests"`, `-m "Bot API docs"`. - Done when: all pass; AC1–AC8 have passing tests. +#### Slice 3c: the retire rule (behavioural; ships with the first version that drops a preset directory) + +- Size and risk: small · high (start-up) · ~100 lines with tests +- Files: src/Simplex/Chat/Operators.hs (rule 2 in the pass), tests/OperatorTests.hs, tests/ChatTests/Directories.hs. +- Change: section 4.2's retire rule, in the version whose presets first drop a directory (a pull request of its own then, or of that version's preset change). +- Tests: retire (a preset its operator no longer lists comes back `preset = False`, `domains = []`, `enabled = False`, its id); a turned-off preset whose address another operator now presets is adopted, not retired; the merge: a retired row appears once, in the custom group, and stays so on a second pass; the property extended: never a retired row at any preset's address; the restart test extended: restart with a config whose operator presets none: the preset is under Your servers, turned off; restart with the first config: back under the operator, still off. +- Verify and done when: as slice 3b, with AC2b passing. + ### Pull request 2: iOS (~1,550 lines; not compiled in the container) #### Slice 5: types and settings state (structural) @@ -287,38 +296,44 @@ The smallest useful first release is pull requests 1, 2 and 3: without the setti #### Slice 6a: adding a directory, boards 11–12 (behavioural) - Size and risk: medium · low · ~250 lines -- Files: DirectoryView.swift (new: `DirectoryView`, `NewDirectoryView`, `DirectoryViewLink` showing "Unnamed directory" when the name is empty, `addDirectory`, the two-step test), ProtocolServersView.swift (Your servers' section with delete, Add server's "Directory", `TestServersButton` covering directories), built from `ChatRelayView`'s shared pieces. +- Files: DirectoryView.swift (new: `DirectoryView`, `NewDirectoryView`, `DirectoryViewLink` showing "Unnamed directory" when the name is empty, `addDirectory`, the two-step test), ProtocolServersView.swift (Your servers' section with delete, Add server's "Directory"), built from `ChatRelayView`'s shared pieces. - Change: section 4.3 for Your servers and New directory. -#### Slice 6b: the operator's directories, board 10 (behavioural) +#### Slice 6b: the operator's directories and testing them, board 10 (behavioural) -- Size and risk: small · low · ~200 lines -- Files: OperatorView.swift (the section, preset `DirectoryView`, `TestServersButton` covering directories), NetworkAndServers.swift (the "No directory is in use" note). -- Change: section 4.3 for the operator screen and the note. +- Size and risk: small · low · ~220 lines +- Files: OperatorView.swift (the section, shown also while the operator is off; preset `DirectoryView`; the call of `TestServersButton` with the group's directories), ProtocolServersView.swift (`TestServersButton` covering directories, a binding per list, and its call in Your servers), NetworkAndServers.swift (the "No directory is in use" note). +- Change: section 4.3 for the operator screen, "Test servers" and the note; `TestServersButton` changes here with both its callers, so each slice compiles. -#### Slice 7a: the sender takes an address and leaves the cooperative pool (structural) +#### Slice 7a: the sender takes a user and an address and leaves the cooperative pool (structural) -- Size and risk: small · low · ~60 lines -- Files: SimpleXAPI.swift (`apiSearchDirectory(address:…)` running the blocking call on a dispatch queue), DirectorySearchView.swift (passes the constant for now). +- Size and risk: small · low · ~70 lines +- Files: SimpleXAPI.swift (`apiSearchDirectory(userId:address:…)` running the blocking call on a dispatch queue; `getUserServers(userId:)`), DirectorySearchView.swift (passes the current user and the constant for now). - Change: no visible change; one directory still. -#### Slice 7b: search across directories, boards 14–15 (behavioural) +#### Slice 7b1: sending to every directory, in order (behavioural) -- Size and risk: large · high (who receives the search text) · ~350 lines -- Files: DirectorySearch.swift (`DIRECTORY_SERVICE_LINK` removed; the 50-entry cap; counts summed saturating), DirectorySearchView.swift (`OnlineSearchModel` per directory; the fresh servers read before each action, with the user id, remote host and generation captured and the user id passed to every request; the hold rule with its 15 s deadline, the earliest-wins rule and Cancel; footer rows once the load completes; load row text), ChatListView.swift and NewChatMenuButton.swift (footer rows; the row's visibility value and its refresh points), the specs and product docs that describe search. -- Change: section 4.4 except consent and the submenu; consent stays today's single switch until 7c, applied to every directory the action contacts. +- Size and risk: medium · high (who receives the search text) · ~220 lines +- Files: DirectorySearch.swift (`DIRECTORY_SERVICE_LINK` removed; the 50-entry cap), DirectorySearchView.swift (`OnlineSearchModel` per directory; the capture of user id, remote host and generation, the servers read for that user and the check before sending; the hold rule with its 15 s deadline; the earliest-wins rule), ChatListView.swift and NewChatMenuButton.swift (the row's visibility value and its refresh points), NetworkAndServers.swift (`saveServers` refreshes that value). +- Change: section 4.4's which directories, sending, state and order; consent stays today's single switch until 7c, applied to every directory the action contacts. Until 7b2 the footer and Cancel work as today over all directories: one retry row when any failed (retrying the failed ones), the load row when any has more, Cancel ignoring every answer still to come. - Pushed together with slice 7d (the workflow's rewrite of the constant fails once the constant is gone). +#### Slice 7b2: the footer across directories, boards 14–15 (behavioural) + +- Size and risk: small · low · ~130 lines +- Files: DirectorySearch.swift (counts summed saturating), DirectorySearchView.swift (per-directory retry rows, the load row's text and when the footer shows, Cancel with several directories), ChatListView.swift and NewChatMenuButton.swift (the footer rows), the specs and product docs that describe search. +- Change: section 4.4's footer and Cancel. + #### Slice 7c: consent naming every directory, board 13 (behavioural) - Size and risk: medium · high (privacy) · ~200 lines -- Files: DirectorySearchView.swift (`confirmDirectorySearch` with the two alerts, the remembered preset addresses, the one-time step if open question 3 is answered as recommended), SettingsView.swift (the key for the remembered addresses), PrivacySettings.swift (turning the switch off forgets them). +- Files: DirectorySearchView.swift (`confirmDirectorySearch` with the two alerts, the remembered preset addresses, the one-time step if open question 3 is answered as recommended), SettingsView.swift (the keys for the remembered addresses and the one-time step), PrivacySettings.swift (turning the switch off forgets them), NetworkAndServers.swift (`saveServers`, which every save path calls, including the submenu's sheet: it forgets addresses that are not an enabled preset of the saved profile). - Change: section 4.4's consent. #### Slice 7d: CI with two directories (CI only, kind 2) - Size and risk: medium · low · ~250 lines -- Files: scripts/ci/ios-sim/backend.sh (two directory services, "CI Directory A" and "CI Directory B" (`--service-name` and the bot's display name, today both "SimpleX Directory", backend.sh:21), overlapping listings, one shared channel whose stored description B's database changes with sqlite3 while B is stopped; `directory-stop` and `directory-start` per directory), scripts/ci/ios-sim/smoke-test.sh (AC12's steps in order: the second alert with the preset on and Cancel; the preset turned off before anything else, "Test servers" included; A then B added in Settings; the checks in both layouts), device-test-plan.md (the one-time step, forgetting on switch-off), claude-prompt.md, .github/workflows/ios-sim.yml (the step that rewrites the constant removed). +- Files: scripts/ci/ios-sim/backend.sh (two directory services, "CI Directory A" and "CI Directory B" (`--service-name` and the bot's display name, today both "SimpleX Directory", backend.sh:21), overlapping listings, one shared channel whose `group_profiles.short_descr` B's database changes with sqlite3 while B is stopped, checked again after B restarts; `directory-stop` and `directory-start` per directory), scripts/ci/ios-sim/smoke-test.sh (AC12's steps in order: the second alert with the preset on and Cancel; the preset turned off before anything else, "Test servers" included; the switch turned off; A then B added in Settings; the checks in both layouts), device-test-plan.md (the one-time step; forgetting on switch-off and on turning a preset off; with SimpleX Chat's servers off, its Directories section shows, "Search in Directory" shows, and the alert names SimpleX Directory, answered with Cancel only), claude-scope (the runner cases only, not the device ones), claude-prompt.md (never turn the SimpleX Directory preset on and never tap Search in the second alert: both reach the live directory), .github/workflows/ios-sim.yml (the step that rewrites the constant removed). - Change: AC12 on iOS. #### Slice 8: the Directories submenu, boards 16–17 (behavioural) @@ -331,9 +346,9 @@ Verification for slices 5–8: the ios-sim workflow builds and runs the scripted ### Pull request 3: Kotlin (Android and desktop; ~1,450 lines) -#### Slices 9, 10a, 10b, 11a–11d, 12: slices 5, 6a, 6b, 7a–7d and 8 mirrored file for file +#### Slices 9, 10a, 10b, 11a, 11b1, 11b2, 11c, 11d, 12: slices 5, 6a, 6b, 7a, 7b1, 7b2, 7c, 7d and 8 mirrored file for file -Same files under apps/multiplatform/common/src/commonMain/kotlin/chat/simplex/common/ (model/SimpleXAPI.kt, model/DirectorySearch.kt, views/usersettings/networkAndServers/*, views/chatlist/DirectorySearchView.kt, ChatListView.kt, views/newchat/NewChatSheet.kt, views/usersettings/PrivacySettings.kt), strings in moko-resources, specs and product docs; CI slice 11d covers scripts/ci/desktop-linux (backend.sh, smoke-test.sh, claude-search.md), the Android emulator test and .github/workflows/desktop-linux.yml. Differences from Swift, each with its reason: slice 11a only takes the address (Kotlin already sends on `Dispatchers.IO`); slice 11b sums counts in `Long` and saturates (today's `Int` would overflow, DirectorySearch.kt:85), checked by review; the one-hand list uses `reverseLayout`, so the menu needs no preview handling. Verified on the desktop-linux and android-emu workflows. +Same files under apps/multiplatform/common/src/commonMain/kotlin/chat/simplex/common/ (model/SimpleXAPI.kt, model/DirectorySearch.kt, views/usersettings/networkAndServers/*, views/chatlist/DirectorySearchView.kt, ChatListView.kt, views/newchat/NewChatSheet.kt, views/usersettings/PrivacySettings.kt), strings in moko-resources, specs and product docs; CI slice 11d covers scripts/ci/desktop-linux (backend.sh, smoke-test.sh, claude-search.md), the Android emulator test and .github/workflows/desktop-linux.yml. Differences from Swift, each with its reason: slice 11a only takes the address (Kotlin already sends on `Dispatchers.IO`); slice 11b2 sums counts in `Long` and saturates (today's `Int` would overflow, DirectorySearch.kt:85), checked by review; slice 10b's `TestServersButton` parameters have defaults, so it could land earlier, but keeps 6b's shape; the one-hand list uses `reverseLayout`, so the menu needs no preview handling. Verified on the desktop-linux and android-emu workflows. ### Pull request 4: the directory answers MCP (base the search branch, or master after #7588; ~250 lines; optional) @@ -357,8 +372,8 @@ Same files under apps/multiplatform/common/src/commonMain/kotlin/chat/simplex/co ## 8. Risks -- Riskiest assumption: the start-up pass and merge add the preset to every existing user without touching their servers or relays, keep a turned-off preset off, adopt a user's row at a preset's address once, and retire a dropped preset without a start-up error. Slice 3a's pure and property tests and slice 3b's merge, invariant and restart tests check it before anything depends on it. -- Retiring fails safe but silently: a version that presets fewer directories (an older or newer binary on the same database) turns the others off (section 4.2). +- Riskiest assumption: the start-up pass and merge add the preset to every existing user without touching their servers or relays, keep a turned-off preset off, and adopt a user's row at a preset's address without a duplicate or a start-up error. Slice 3a's pure and property tests and slice 3b's merge, invariant and restart tests check it before anything depends on it; slice 3c's tests do the same for retiring when it ships. +- Retiring fails safe but silently: from the version that ships it, a binary that presets fewer directories (an older or newer one on the same database) turns the others off (section 4.2); shipping it only when a preset is first dropped postpones this. - The 5-tuple: every caller of `getUserServers` changes; the compiler finds them all, in slice 1, with no behaviour change. - Remote pairs: only a JSON test covers a newer desktop with an older phone in the container; the older-desktop bypass exists if search ships first (section 5.3). - CI for pull requests 2 and 3 (two directory services in the backends; adding them through Settings with the `ui` helpers; turning the preset off before any search or "Test servers" press, and tapping only Cancel on the alert while it is on, so CI never reaches the live directory) is new scripted UI work; desktop's text-recognition driver is the slower part. Pull request 2's run should rehearse it first. @@ -369,16 +384,16 @@ Same files under apps/multiplatform/common/src/commonMain/kotlin/chat/simplex/co 1. **This design or the service layer first** (ep). Recommended: this design now. It is drawn, needs no protocol change for search, reuses every layer relays built, and leaves storage, search and consent in place if a services list comes later (~1,500 lines, section 4.6.3). If not: the services design comes first, and the search side of this plan (section 4.4) carries over. 2. **The core pull request to master right after question 1** (ep). Recommended: yes, if question 1 is this design, and before directory search is released if timing allows: then every phone with directory search already answers with its directories, which removes question 3's first cost. It is invisible to users and settles storage, presets and compatibility before the apps start. Merging it into a release freezes the migration and writes a SimpleX Directory row into every profile, so a later change of storage (a services table) is a second migration. If not: it waits, and the apps' work waits for it. -3. **Release order** (ep and the user). Recommended: ship directory search when it is ready, and several directories in a later release. Shipping them together delays directory search, already reviewed, by the core and both apps' pull requests (~3,900 lines and two runner passes), while several directories add little until other directories exist. Costs of shipping separately, accepted: a newer desktop paired with a phone of the search-only release can't search through it (gone if the core lands before that release, question 2); an older desktop paired with a newer phone sends its searches to SimpleX Directory even if the phone's user turned it off (section 5.3); a user who turned the SimpleX Chat operator's servers off can search in the search-only release (it ignores operators) and loses search on the upgrade, with only Network & servers' note saying why (question 9); the consent text changes in two releases; the one-time consent step is needed. If together: none of these, and directory search waits. +3. **Release order** (ep and the user). Recommended: ship directory search when it is ready, and several directories in a later release. Shipping them together delays directory search, already reviewed, by the core and both apps' pull requests (~3,900 lines and two runner passes), while several directories add little until other directories exist. Costs of shipping separately, accepted: a newer desktop paired with a phone of the search-only release can't search through it (gone if the core lands before that release, question 2); an older desktop paired with a newer phone sends its searches to SimpleX Directory even if the phone's user turned it off (section 5.3); the consent text changes in two releases; the one-time consent step is needed. If together: none of these, and directory search waits. 4. **Consent with several directories.** Recommended: section 4.4 (one switch; remembered operator directories only; a user-added directory agreed to by adding it; a second alert for a preset not yet agreed to). If you'd rather keep the user's own directories in the alerts too: their addresses would sit in app preferences outside the encrypted database, including those of hidden profiles; or a consent column in the core, per profile, decided before the core migration ships. 5. **The load row with two or more directories.** Recommended: "Load more · about N left" (the wording you chose for one directory, with "about"). If the board's "Load up to 20 more · about 34 left": the apps read `pageSize` (sent since the search work) and add the directories' page sizes. 6. **Order and arrival.** Recommended: section 4.4 (settings order; a directory's rows appear once every earlier one has answered, failed or passed 15 s; the earliest directory's entry for a shared link; footer rows once the load is complete). Alternatives: arrival order (no waiting, but the fastest directory's rows and claims come first, and the order changes from search to search); placing rows in settings order as they arrive (rows move under the user's finger); waiting for all (the slowest holds every result for up to 15 s, the same bound without showing anyone's results early). 7. **"Test directory".** Recommended: in the apps, the connect plan for the name plus one empty-text search. If you'd rather have a core command like "Test relay": the core learns the search request's shape. 8. **No directory in use.** Recommended: "Search in Directory" hides, the Search key searches nothing beyond your chats, and Network & servers says so. If not: the row stays and a tap says no directory is in use. -9. **The operator switch and directories.** Recommended: turning an operator's servers off also stops its directories, as it stops its relays for new channels. Costs: a directory the user added, which a later version presets under an operator the user turned off, stops being searched and is hidden with that operator's section until it is turned on; and a user who turned the SimpleX Chat operator off but searched in the search-only release loses search on the upgrade (question 3). If a directory's own switch alone decides: an operator turned off for its servers still receives searches. +9. **The operator switch and directories** (ep: SimpleX Directory's terms are part of SimpleX Chat's conditions, PRIVACY.md:124-126, so this is a policy call). Recommended: a directory's own switch decides; an operator's "Use servers" switch is about its servers for messages, and the operator screen shows its Directories section also while its servers are off. Then a new user who chose only Flux at onboarding, or someone running their own servers with both operators off, still finds SimpleX Directory, named in the consent alert before their first search; cost: a user who turned an operator's servers off can still send searches to its directory once they agree to it in the alert, and its section shows on a screen whose relays are hidden. If turning an operator's servers off also stops its directories (as channel creation skips a disabled operator's relays; its argument: SimpleX Directory's terms are part of the operator's conditions, which the user accepts by turning it on): every user without the SimpleX Chat operator, Flux-only users and self-hosters included, has no directory search; they can't add SimpleX Directory themselves (its address collides with the hidden preset row); the only way back is turning SimpleX Chat's servers on, which can mean accepting its conditions and using its servers again; a user who searched in the search-only release with that operator off loses search on the upgrade; and a directory a later version presets under a turned-off operator stops being searched. 10. **Retry rows.** Recommended: as drawn (one per failed directory, the others' results stay). 11. **A cap on directories searched at once.** Recommended: none for now (each one is added by the user; iOS requests are off the cooperative pool). If wanted: search the first N in settings order and say so in Settings. -12. **A directory a later version no longer presets.** Recommended: keep it, turned off, under Your servers (section 4.2, rule 2), with its stated costs (a binary that presets fewer turns directories off silently). If deleting is preferred: a store delete outside the preset guard, and a later re-preset turns it back on without asking. +12. **A directory a later version no longer presets, and when that rule ships.** Recommended: keep it, turned off, under Your servers (section 4.2, rule 2), shipped with the first version that drops a preset (slice 3c), not now: until then it would do nothing and only start its cost (a binary with the rule that presets fewer directories turns the others off silently). The adopt rule ships now: it costs nothing and guards a start-up failure in the first version that presets an address a user may have added. If deleting is preferred: a store delete outside the preset guard at start-up, which the app's remembered addresses don't see, so a later re-preset turns it back on and it can receive searches without a question. If retire ships now: written and tested once, and its cost starts with the first release. 13. **When the directory answers ep's MCP envelope** (ep). Recommended: together with the core's `protocols` declarations from his draft; before them nothing can find or call it over Service RPC. If now: ~250 lines, a proof of the envelope on a real service. 14. **Shipping while searches can be linked** (ep). Recommended: ship. The alerts promise only "each search separately and without your profile", which holds; colluding directories can link one search by its text and time, and a self-hosting user's searches can be linked by their reply server, as with one directory today. The simplexmq fix (reply queues on their own transport session) is proposed and unplanned. If not: several directories wait for it.