mirror of
https://github.com/simplex-chat/simplex-chat.git
synced 2026-11-22 15:38:23 +00:00
* core, ui: validate short link type, show contact info for long addresses and rejected contacts (#8184) * tests: short link type, QR code capacity and contact statuses * core: validate the type of short links * android, desktop: show contact info for addresses above QR code capacity * ui: show chats with rejected contact requests * tests: short link types with group, channel and contact data * core: validate short link type against link data * tests: fix short link data test * tests: group data in invitation link before its first use * tests: connect via address in contact profile with group link * core: connect via contact address only with address links * core: parse short link data once, by link type * core: types of short link data parser * core: test short link types and contact rejection, record app fixtures * android, desktop: test contact info, link alerts and rejected contacts * ios: test invalid link alerts and rejected contacts in the chat list * desktop: contact info tests find texts without app resources * ui: core responses of invalid link plans and a rejected contact request * desktop: contact info tests provide the app bar handler * core: normalise file and message keys in app test fixtures * core: normalise deletion times in app test fixtures * desktop: test that a QR code above capacity is not shared on tap * core: test relay link checks and known channel refresh from link data * libs: nodejs 7.1.4 (#8263) * nodejs: live test reads the updated item under the key of 7.1 and 7.2 cores * libs: nodejs 7.1.4 * mcp: session states coloured by what they need, green for working (#8265) * mcp: session states in the colours of what they need: green working, orange waiting in the chat, red idle * mcp: the space after waiting for you in the tests * mcp: a session with a background command shows working in the background * mcp: a readable README, and the roadmap lines of 0.3.0 (#8264) * mcp: readme by what the plugin does for its users, roadmap lines of 0.3.0 * mcp: almost every waiting state * mcp: reply, edit and react in one line * mcp: flat layout without the avatar note * mcp: sessions use flat layout * mcp: shorter lines on answers, the Main group and approvals * mcp: approvals line without the taps * mcp: live stream line * mcp: status line * mcp: shorter lines * mcp: 0.3.1 with simplex-chat 7.1.4 (#8266) * docs: testing approaches of the codebase (#8255) * docs: app changes come with app tests on the core's recorded responses (#8252) * ci: decide the library claim from the written releases (#8137) * ci: test the library claim when reads return the previous releases * ci: decide the library claim from the written releases * tests: record name fixtures after the channel join settles (#8250) * core, ci: update sqlcipher-simple to 0.4.19.0 and simplexmq (#8270) * update sqlcipher-simple to 0.4.19.0 * sqlcipher-simple from simplex-chat/sqlcipher-simple * update simplexmq * ci: restore source repositories only for the same cabal.project * ci: keep restored source checkouts only at their pinned commits * android, desktop, ios: copy markdown files with line breaks, and the whole file with one button (#8269) * android, desktop: test copying lines from the markdown text view * android, desktop: keep line breaks in text copied from markdown files * android, desktop: check the last copied text in the markdown view test * android, desktop, ios: button to copy the whole markdown file * android, desktop: test the markdown file copy button * android, desktop, ios: ask before copying markdown files over 128 KB * android, desktop, ios: title the markdown copy alerts File too large * ios: test the markdown file copy button * android, desktop, ios: preview the markdown file view and test the copy limit * android, desktop, ios: snapshot references of the markdown file view * docs: names conditions describe upgradeable registry contracts (#8277) * docs: names conditions describe upgradeable registry contracts * docs: names conditions without the immutability after stabilization --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> * ci: reproducible Linux builds of 7.1 releases (#8276) * ci: plan reproducible Linux builds of 7.1 releases * ci: reproduce Linux builds with the cabal store path of releases * ci: reproduce releases with the current reproduce script * ci: build in the image with the cabal directory of releases * ci: reproduce releases on the Ubuntu 24.04 runner * ci: correct the failure history and rollout in the plan --------- Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> * website: remove the webinar promotion (#8281) * website: remove the webinar invitation from the home page * website: redirect the livestream page to Wefunder * mcp server: recall the chat history in a new agent session (#8141) * mcp server: recall the chat history in a new agent session * docs: chat history in receive for a new agent session * mcp server: type the history page for the compiler * mcp server: recall the chat history after a compaction too * docs: history plan changes with the message helpers of master Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75 * docs: drop anchor remapping from the docs table of the history plan Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75 --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> * core: prevent stalled join of channel subscribers to a new relay (#8239) * core: save relay connection before requesting relay link data * core: share relay connection creation between join and retry * tests: connect to a re-added relay when its connection is saved late * tests: delay only the relay connection record, not the command * ci, tests: run all tests in CI (#8238) * bots, tests: run broadcast bot test in CI * tests: test moderation that arrives before the message * ci, tests: run all tests in CI * tests: update re-enabled tests, restore query plans, PostgreSQL schema from CI * tests: remove CI skip helpers no longer used * tests: expect XFTP AUTH error of cancelled standalone file * tests: update query plans from CI * tests: update query plans from CI * tests: expect group history and member contacts in async group test * tests: accept subscription summary after first events on restart * tests: make AUTH and client timeout direct tests deterministic * tests: keep bob's badge clock before renewal until the move back * tests: keep the quota of 2 messages only on the member's receive queue * tests: keep query plans files with repeated entries of one query * tests: update query plans from CI * tests: stop the host before a member requests to join with a badge * tests: remove repeated entries of one query from query plans files * tests: accept the refusal of an unknown remote host after its session code * ci: state when the query plans test fails * tests: skip multicast discovery on macOS CI runners * tests: accept a remote host started with one local address * tests: restore the client timeout test except its zero timeout * tests: fail when a remote host prints another line than its other addresses * ci: name the tests that overwrite generated files * tests: move the badge clock after the restoring device arms its wake * tests: keep both members on the quota server in the member inactivity test * tests: accept subscription summaries in any order in the async group test * tests: update query plans from CI * tests: quota of 2 only on bob's queues in member inactivity test * tests: connect to a deleted link after its queue is deleted * core: save relay connection before requesting relay link data * core: share relay connection creation between join and retry * tests: connect to a re-added relay when its connection is saved late * tests: delay only the relay connection record, not the command * tests: record API fixtures on SQLite only * core: update query plans * ci: run PostgreSQL tests with the steps of build-linux * desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts (#8288) * desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts * desktop: record time offsets and snapshots rendered with Skia m138 * desktop: name the time offset test, skip only a missing fc-cache, update plan --------- Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> * core, directory: stop cancelled actions instead of continuing them (#8279) * ci: retry npm install once after a failure (#8280) * core: query plans of the store group tests that master now verifies --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> Co-authored-by: sh-agents <justcheckingifitwillwork@gmail.com> Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> Co-authored-by: brenzi-agent <brenzi-agent@brenzi.ch> Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
212 lines
9.0 KiB
Haskell
212 lines
9.0 KiB
Haskell
{-# LANGUAGE LambdaCase #-}
|
|
{-# LANGUAGE NamedFieldPuns #-}
|
|
{-# LANGUAGE OverloadedStrings #-}
|
|
{-# LANGUAGE ScopedTypeVariables #-}
|
|
|
|
module SchemaDump where
|
|
|
|
import ChatTests.DBUtils
|
|
import Control.Concurrent.STM
|
|
import Control.DeepSeq
|
|
import qualified Control.Exception as E
|
|
import Control.Monad (unless, void)
|
|
import Data.List (dropWhileEnd)
|
|
import qualified Data.Map.Strict as M
|
|
import Data.Maybe (fromJust, isJust)
|
|
import Data.Text (Text)
|
|
import qualified Data.Text as T
|
|
import qualified Data.Text.IO as T
|
|
import Database.SQLite.Simple (Only (..), Query (..))
|
|
import Simplex.Chat.Options.SQLite (chatDBFunctions)
|
|
import Simplex.Chat.Store (createChatStore)
|
|
import qualified Simplex.Chat.Store as Store
|
|
import Simplex.Messaging.Agent.Env.SQLite (createAgentStore)
|
|
import Simplex.Messaging.Agent.Store.Common (withConnection)
|
|
import Simplex.Messaging.Agent.Store.DB (TrackQueries (..))
|
|
import qualified Simplex.Messaging.Agent.Store.DB as DB
|
|
import Simplex.Messaging.Agent.Store.Interface
|
|
import qualified Simplex.Messaging.Agent.Store.SQLite.Migrations as Migrations
|
|
import Simplex.Messaging.Agent.Store.Shared (Migration (..), MigrationConfig (..), MigrationConfirmation (..), MigrationsToRun (..), toDownMigration)
|
|
import Simplex.Messaging.Util (ifM, tshow, whenM)
|
|
import System.Directory (doesFileExist, removeFile)
|
|
import System.Process (readCreateProcess, shell)
|
|
import Test.Hspec
|
|
|
|
testDB :: FilePath
|
|
testDB = "tests/tmp/test_chat.db"
|
|
|
|
testAgentDB :: FilePath
|
|
testAgentDB = "tests/tmp/test_agent.db"
|
|
|
|
appSchema :: FilePath
|
|
appSchema = "src/Simplex/Chat/Store/SQLite/Migrations/chat_schema.sql"
|
|
|
|
appLint :: FilePath
|
|
appLint = "src/Simplex/Chat/Store/SQLite/Migrations/chat_lint.sql"
|
|
|
|
appChatQueryPlans :: FilePath
|
|
appChatQueryPlans = "src/Simplex/Chat/Store/SQLite/Migrations/chat_query_plans.txt"
|
|
|
|
appAgentQueryPlans :: FilePath
|
|
appAgentQueryPlans = "src/Simplex/Chat/Store/SQLite/Migrations/agent_query_plans.txt"
|
|
|
|
testSchema :: FilePath
|
|
testSchema = "tests/tmp/test_agent_schema.sql"
|
|
|
|
schemaDumpTest :: Spec
|
|
schemaDumpTest = do
|
|
it "verify and overwrite schema dump" testVerifySchemaDump
|
|
it "verify .lint fkey-indexes" testVerifyLintFKeyIndexes
|
|
it "verify schema down migrations" testSchemaMigrations
|
|
it "verify strict tables" testVerifyStrict
|
|
|
|
testVerifySchemaDump :: IO ()
|
|
testVerifySchemaDump = do
|
|
savedSchema <- ifM (doesFileExist appSchema) (readFile appSchema) (pure "")
|
|
savedSchema `deepseq` pure ()
|
|
void $ createChatStore (DBOpts testDB chatDBFunctions "" False True TQOff) (MigrationConfig MCError Nothing)
|
|
getSchema testDB appSchema `shouldReturn` savedSchema
|
|
removeFile testDB
|
|
|
|
testVerifyLintFKeyIndexes :: IO ()
|
|
testVerifyLintFKeyIndexes = do
|
|
savedLint <- ifM (doesFileExist appLint) (readFile appLint) (pure "")
|
|
savedLint `deepseq` pure ()
|
|
void $ createChatStore (DBOpts testDB chatDBFunctions "" False True TQOff) (MigrationConfig MCError Nothing)
|
|
getLintFKeyIndexes testDB "tests/tmp/chat_lint.sql" `shouldReturn` savedLint
|
|
removeFile testDB
|
|
|
|
testSchemaMigrations :: IO ()
|
|
testSchemaMigrations = do
|
|
let noDownMigrations = dropWhileEnd (\Migration {down} -> isJust down) Store.migrations
|
|
Right st <- createDBStore (DBOpts testDB chatDBFunctions "" False True TQOff) noDownMigrations (MigrationConfig MCError Nothing)
|
|
mapM_ (testDownMigration st) $ drop (length noDownMigrations) Store.migrations
|
|
closeDBStore st
|
|
removeFile testDB
|
|
whenM (doesFileExist testSchema) $ removeFile testSchema
|
|
where
|
|
testDownMigration st m = do
|
|
putStrLn $ "down migration " <> name m
|
|
let downMigr = fromJust $ toDownMigration m
|
|
schema <- getSchema testDB testSchema
|
|
Migrations.run st Nothing True $ MTRUp [m]
|
|
schema' <- getSchema testDB testSchema
|
|
unless (name m `elem` skipComparisonForUpMigrations) $
|
|
schema' `shouldNotBe` schema
|
|
Migrations.run st Nothing True $ MTRDown [downMigr]
|
|
unless (name m `elem` skipComparisonForDownMigrations) $ do
|
|
schema'' <- getSchema testDB testSchema
|
|
schema'' `shouldBe` schema
|
|
Migrations.run st Nothing True $ MTRUp [m]
|
|
schema''' <- getSchema testDB testSchema
|
|
schema''' `shouldBe` schema'
|
|
|
|
testVerifyStrict :: IO ()
|
|
testVerifyStrict = do
|
|
Right st <- createDBStore (DBOpts testDB chatDBFunctions "" False True TQOff) Store.migrations (MigrationConfig MCError Nothing)
|
|
withConnection st (`DB.query_` "SELECT name FROM sqlite_master WHERE type = 'table' AND name != 'sqlite_sequence' AND sql NOT LIKE '% STRICT'")
|
|
`shouldReturn` ([] :: [Only Text])
|
|
|
|
skipComparisonForUpMigrations :: [String]
|
|
skipComparisonForUpMigrations =
|
|
[ -- schema doesn't change
|
|
"20250129_delete_unused_contacts"
|
|
]
|
|
|
|
skipComparisonForDownMigrations :: [String]
|
|
skipComparisonForDownMigrations =
|
|
[ -- on down migration msg_delivery_events table moves down to the end of the file
|
|
"20230504_recreate_msg_delivery_events_cleanup_messages",
|
|
-- on down migration idx_chat_items_timed_delete_at index moves down to the end of the file
|
|
"20230529_indexes",
|
|
-- table and index definitions move down the file, so fields are re-created as not unique
|
|
"20230914_member_probes",
|
|
-- on down migration idx_connections_via_contact_uri_hash index moves down to the end of the file
|
|
"20231019_indexes",
|
|
-- table and indexes move down to the end of the file
|
|
"20231215_recreate_msg_deliveries",
|
|
-- on down migration idx_msg_deliveries_agent_ack_cmd_id index moves down to the end of the file
|
|
"20240313_drop_agent_ack_cmd_id",
|
|
-- sequence table moves down to the end of the file
|
|
"20241023_chat_item_autoincrement_id",
|
|
-- indexes move down to the end of the file
|
|
"20241125_indexes",
|
|
-- indexes move down to the end of the file
|
|
"20250130_indexes",
|
|
-- index moves down to the end of the file
|
|
"20250227_member_acceptance",
|
|
-- index moves down to the end of the file
|
|
"20250721_indexes",
|
|
-- indexes move down to the end of the file
|
|
"20250922_remove_unused_connections",
|
|
-- group_member_intros table moves down to the end of the file
|
|
"20251128_migrate_member_relations",
|
|
-- on down migration single_sender_group_member_id column and its index
|
|
-- are re-added at the end of the table / file (ALTER TABLE ADD COLUMN
|
|
-- appends; CREATE INDEX appends).
|
|
"20260529_delivery_job_senders",
|
|
-- group_domain is removed
|
|
"20260603_simplex_name",
|
|
"20261018_autoincrement_ids"
|
|
]
|
|
|
|
getSchema :: FilePath -> FilePath -> IO String
|
|
getSchema dbPath schemaPath = do
|
|
void $ readCreateProcess (shell $ "sqlite3 " <> dbPath <> " '.schema --indent' > " <> schemaPath) ""
|
|
sch <- readFile schemaPath
|
|
sch `deepseq` pure sch
|
|
|
|
getLintFKeyIndexes :: FilePath -> FilePath -> IO String
|
|
getLintFKeyIndexes dbPath lintPath = do
|
|
void $ readCreateProcess (shell $ "sqlite3 " <> dbPath <> " '.lint fkey-indexes' > " <> lintPath) ""
|
|
lint <- readFile lintPath
|
|
lint `deepseq` pure lint
|
|
|
|
saveQueryPlans :: SpecWith TestParams
|
|
saveQueryPlans = it "verify and overwrite query plans" $ \TestParams {chatQueryStats, agentQueryStats} -> do
|
|
(chatSavedPlans, chatSavedPlans') <-
|
|
updatePlans
|
|
appChatQueryPlans
|
|
chatQueryStats
|
|
(createChatStore (DBOpts testDB chatDBFunctions "" False True TQOff) (MigrationConfig MCError Nothing))
|
|
(\db -> do
|
|
DB.execute_ db "CREATE TABLE IF NOT EXISTS temp_conn_ids (conn_id BLOB)"
|
|
DB.execute_ db "CREATE TABLE IF NOT EXISTS temp_delete_members (contact_profile_id INTEGER, member_profile_id INTEGER, local_display_name TEXT)"
|
|
)
|
|
(agentSavedPlans, agentSavedPlans') <-
|
|
updatePlans
|
|
appAgentQueryPlans
|
|
agentQueryStats
|
|
(createAgentStore (DBOpts testAgentDB [] "" False True TQOff) (MigrationConfig MCError Nothing))
|
|
(const $ pure ())
|
|
chatSavedPlans' == chatSavedPlans `shouldBe` True
|
|
agentSavedPlans' == agentSavedPlans `shouldBe` True
|
|
removeFile testDB
|
|
removeFile testAgentDB
|
|
where
|
|
updatePlans plansFile statsSel createStore prepareStore = do
|
|
savedPlans <- ifM (doesFileExist plansFile) (T.readFile plansFile) (pure "")
|
|
savedPlans `deepseq` pure ()
|
|
recorded <- readTVarIO statsSel
|
|
Right st <- createStore
|
|
plans' <- withConnection st $ \db -> do
|
|
void $ prepareStore db
|
|
M.traverseMaybeWithKey (getQueryPlan db) $ M.union (True <$ recorded) (savedQueries savedPlans)
|
|
let savedPlans' = T.unlines $ M.elems plans'
|
|
T.writeFile plansFile savedPlans'
|
|
pure (savedPlans, savedPlans')
|
|
savedQueries = M.fromList . map savedQuery . drop 1 . T.splitOn "\nQuery: " . ("\n" <>)
|
|
savedQuery entry = case T.breakOn "\nError: " entry of
|
|
(_, "") -> (Query $ fst $ T.breakOn "\nPlan:\n" entry, False)
|
|
(q, _) -> (Query q, True)
|
|
getQueryPlan :: DB.Connection -> Query -> Bool -> IO (Maybe Text)
|
|
getQueryPlan db q keepError =
|
|
fmap (("Query: " <> fromQuery q) <>) . result keepError <$> E.try (DB.query_ db $ "explain query plan " <> q)
|
|
result keepError = \case
|
|
Right r -> Just $ "\nPlan:\n" <> T.unlines (map planDetail r)
|
|
Left (e :: E.SomeException)
|
|
| keepError -> Just $ "\nError: " <> tshow e <> "\n"
|
|
| otherwise -> Nothing
|
|
planDetail :: (Int, Int, Int, Text) -> Text
|
|
planDetail (_, _, _, detail) = detail
|