simplex-chat

mirror of https://github.com/simplex-chat/simplex-chat.git synced 2026-03-30 18:35:49 +00:00

Files

sh 7c92c7666c scripts/db: update documentation, add pg -> sqlite section (#6177 )

* scripts/db: update doc

* scripts/db: update documentation, add pg2sqlite script

* scripts/db/pg2sqlite: formatiing, code quality, modularity

* scripts/db: parallel pgloader

2025-09-15 14:06:39 +01:00

pg2sqlite.py

scripts/db: update documentation, add pg -> sqlite section (#6177 )

2025-09-15 14:06:39 +01:00

README.md

scripts/db: update documentation, add pg -> sqlite section (#6177 )

2025-09-15 14:06:39 +01:00

sqlite.load

scripts/db: update documentation, add pg -> sqlite section (#6177 )

2025-09-15 14:06:39 +01:00

README.md

Transfer data from SQLite to Postgres database

Decrypt SQLite database if it is encrypted.

Agent:

Open sqlite db:

sqlcipher simplex_v1_agent.db

Set your db password:

PRAGMA key = '<your_password>';

Check if db was successfully decrypted:

SELECT count(*) FROM sqlite_master;

Attach new empty db:

ATTACH DATABASE 'simplex_v1_agent_plaintext.db' AS plaintext KEY '';

Export opened db to attached db as plaintext:

SELECT sqlcipher_export('plaintext');

Deattach the plaintext db:

DETACH DATABASE plaintext;

Chat:

Open sqlite db:

sqlcipher simplex_v1_chat.db

Set your db password:

PRAGMA key = '<your_password>';

Check if db was successfully decrypted:

SELECT count(*) FROM sqlite_master;

Attach new empty db:

ATTACH DATABASE 'simplex_v1_chat_plaintext.db' AS plaintext KEY '';

Export opened db to attached db as plaintext:

SELECT sqlcipher_export('plaintext');

Deattach the plaintext db:

DETACH DATABASE plaintext;

Prepare Postgres database.

Connect to PostgreSQL databse:
```
psql -U postgres -h localhost
```

Create user with password:

CREATE USER simplex WITH ENCRYPTED PASSWORD '123123';

Create database:
```
CREATE DATABASE simplex_v1;
```

Assign permissions:

GRANT ALL PRIVILEGES ON DATABASE simplex_v1 TO simplex;

Prepare database:

You should build the CLI binary from the same TAG as the desktop.
1. Build CLI with PostgreSQL support:
```
cabal build -fclient_postgres exe:simplex-chat
```
  And rename it to:
```
mv simplex-chat simplex-chat-pg
```
2. Execute CLI:
```
./simplex-chat-pg -d "postgresql://simplex:123123@localhost:5432/simplex_v1" --create-schema
```
  Press Ctrl+C when CLI ask for a display name.
This should create simplex_v1_agent_schema and simplex_v1_chat_schema schemas in simplex_v1 database, with migrations tables populated. Some tables would have initialization data - it will be truncated via pgloader command in next step.

Load data from decrypted SQLite databases to Postgres database via pgloader.

Install pgloader and add it to PATH. Run in shell (substitute paths):

export POSTGRES_CONN='postgresql://simplex:123123@localhost:5432/simplex_v1'

And then:

SQLITE_DBPATH='simplex_v1_agent_plaintext.db' \
POSTGRES_SCHEMA='simplex_v1_agent_schema' \
CPU_CORES=$(nproc) WORKERS=$((CPU_CORES - 1)) pgloader --dynamic-space-size 262144 --on-error-stop sqlite.load

SQLITE_DBPATH='simplex_v1_chat_plaintext.db' \
POSTGRES_SCHEMA='simplex_v1_chat_schema' \
CPU_CORES=$(nproc) WORKERS=$((CPU_CORES - 1)) pgloader --dynamic-space-size 262144 --on-error-stop sqlite.load

Update sequences for Postgres tables.

Connect to db:

PGPASSWORD=123123 psql -h localhost -U simplex -d simplex_v1

Execute the following:

For agent:

DO $$
DECLARE
   rec RECORD;
BEGIN
   EXECUTE 'SET SEARCH_PATH TO simplex_v1_agent_schema';

   FOR rec IN
      SELECT
         table_name,
         column_name,
         pg_get_serial_sequence(table_name, column_name) AS seq_name
      FROM
         information_schema.columns
      WHERE
         table_schema = 'simplex_v1_agent_schema'
         AND identity_generation = 'ALWAYS'
   LOOP
      EXECUTE format(
         'SELECT setval(%L, (SELECT MAX(%I) FROM %I))',
         rec.seq_name, rec.column_name, rec.table_name
      );
   END LOOP;
END $$;

For chat:

DO $$
DECLARE
   rec RECORD;
BEGIN
   EXECUTE 'SET SEARCH_PATH TO simplex_v1_chat_schema';

   FOR rec IN
      SELECT
         table_name,
         column_name,
         pg_get_serial_sequence(table_name, column_name) AS seq_name
      FROM
         information_schema.columns
      WHERE
         table_schema = 'simplex_v1_chat_schema'
         AND identity_generation = 'ALWAYS'
   LOOP
      EXECUTE format(
         'SELECT setval(%L, (SELECT MAX(%I) FROM %I))',
         rec.seq_name, rec.column_name, rec.table_name
      );
   END LOOP;
END $$;

Compare number of rows between Postgres and SQLite tables.

PostgreSQL:

For agent:

WITH tbl AS (
   SELECT table_schema, table_name
   FROM information_schema.Tables
   WHERE table_name NOT LIKE 'pg_%'
     AND table_schema IN ('simplex_v1_agent_schema')
)
SELECT
   table_schema AS schema_name,
   table_name,
   (xpath('/row/c/text()', query_to_xml(
      format('SELECT count(*) AS c FROM %I.%I', table_schema, table_name), false, true, ''
   )))[1]::text::int AS records_count
FROM tbl
ORDER BY records_count DESC;

For chat:

WITH tbl AS (
   SELECT table_schema, table_name
   FROM information_schema.Tables
   WHERE table_name NOT LIKE 'pg_%'
     AND table_schema IN ('simplex_v1_chat_schema')
)
SELECT
   table_schema AS schema_name,
   table_name,
   (xpath('/row/c/text()', query_to_xml(
      format('SELECT count(*) AS c FROM %I.%I', table_schema, table_name), false, true, ''
   )))[1]::text::int AS records_count
FROM tbl
ORDER BY records_count DESC;

SQLite:

For agent:

db="simplex_v1_agent_plaintext.db"
sqlite3 "$db" "SELECT name FROM sqlite_master WHERE type='table';" |
while read table; do
   count=$(sqlite3 "$db" "SELECT COUNT(*) FROM \"$table\";")
   echo "$table: $count"
done | sort -k2 -nr | less

For chat:

db="simplex_v1_chat_plaintext.db"
sqlite3 "$db" "SELECT name FROM sqlite_master WHERE type='table';" |
while read table; do
   count=$(sqlite3 "$db" "SELECT COUNT(*) FROM \"$table\";")
   echo "$table: $count"
done | sort -k2 -nr | less

Build and run desktop app with Postgres backend.

Run in shell (paths are from project root):

./scripts/desktop/build-lib-mac.sh arm64 postgres

./gradlew runDistributable -Pdatabase.backend=postgres
# or
./gradlew packageDmg -Pdatabase.backend=postgres

Transfer data from SQLite to Postgres database

Prepare sqlite db:

Download simplex-chat CLI:

You should download the CLI binary from the same TAG as the desktop.

export TAG='v6.4.3.1'
curl -L "https://github.com/simplex-chat/simplex-chat/releases/download/${TAG}/simplex-chat-ubuntu-22_04-x86_64" -o 'simplex-chat'

Run the CLI:
```
./simplex-chat
```
Press Ctrl+C when CLI ask for a display name.

Move database:

mv ~/.simplex/simplex_v1_* ~/.local/share/simplex/

Transfer data:

./pg2sqlite.py --verbose 'postgresql://simplex:123123@localhost:5432/simplex_v1' ~/.local/share/simplex/

Update BLOBs:

sqlite3 simplex_v1_chat.db

UPDATE group_members SET member_role = CAST(member_role as BLOB);
UPDATE user_contact_links SET group_link_member_role = CAST(group_link_member_role AS BLOB) WHERE group_link_member_role is not null;