dandri/simplex-chat
1
0
Fork 0
mirror of https://github.com/simplex-chat/simplex-chat.git synced 2026-03-31 20:36:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f0799ef2a5ff8636794aa438bfcf2fb5d05907a8
simplex-chat/apps/multiplatform/spec/database.md
spaced4ndy a09acda329 multiplatform: product specification (#6655)
2026-02-26 17:54:44 +00:00

394 lines
20 KiB
Markdown
Raw Blame History

# Database & Storage
## Table of Contents
1. [Overview](#1-overview)
2. [Database Files & Paths](#2-database-files--paths)
3. [Haskell Store Modules](#3-haskell-store-modules)
4. [Migrations](#4-migrations)
5. [Database Encryption](#5-database-encryption)
6. [File Storage](#6-file-storage)
7. [Export & Import](#7-export--import)
8. [Source Files](#8-source-files)
---
## 1. Overview
SimpleX Chat uses **two SQLite databases** managed entirely by the Haskell core. Kotlin code **never reads or writes the databases directly** -- all data access goes through the JNI command/response protocol defined in [SimpleXAPI.kt](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt).
The two databases are:
| Database | Suffix | Contents |
|----------|--------|----------|
| Chat database | `_chat.db` | Users, contacts, groups, messages, files metadata, settings |
| Agent database | `_agent.db` | SMP/XFTP agent state: connections, queues, encryption keys, delivery tracking |
Both databases are created and migrated by the `chatMigrateInit` JNI function. The Kotlin layer handles:
- Providing the correct file path prefix (`dbAbsolutePrefixPath`)
- Providing the encryption key
- Interpreting migration results (`DBMigrationResult`)
- Exposing API functions that proxy to Haskell store operations
---
## 2. Database Files & Paths
### Expect Declarations
The common module declares platform-dependent paths as `expect` values in [Files.kt](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt):
```kotlin
expect val dataDir: File // L18
expect val tmpDir: File // L19
expect val filesDir: File // L20
expect val appFilesDir: File // L21
expect val wallpapersDir: File // L22
expect val coreTmpDir: File // L23
expect val dbAbsolutePrefixPath: String // L24
expect val preferencesDir: File // L25
expect val preferencesTmpDir: File // L26
expect val chatDatabaseFileName: String // L28
expect val agentDatabaseFileName: String // L29
expect val databaseExportDir: File // L35
expect val remoteHostsDir: File // L37
```
### Android Actual Values
From [Files.android.kt](../common/src/androidMain/kotlin/chat/simplex/common/platform/Files.android.kt):
| Variable | Value | Notes |
|----------|-------|-------|
| `dataDir` | `androidAppContext.dataDir` | `/data/data/<package>/` |
| `tmpDir` | `getDir("temp", MODE_PRIVATE)` | Private temp directory |
| `filesDir` | `dataDir/files` | Parent for all file storage |
| `appFilesDir` | `filesDir/app_files` | User-visible chat file attachments |
| `wallpapersDir` | `filesDir/assets/wallpapers` | Custom wallpaper images |
| `coreTmpDir` | `filesDir/temp_files` | Haskell core temp directory |
| `dbAbsolutePrefixPath` | `dataDir/files` | Prefix: core appends `_chat.db` / `_agent.db` |
| `chatDatabaseFileName` | `"files_chat.db"` | Full filename: `files_chat.db` |
| `agentDatabaseFileName` | `"files_agent.db"` | Full filename: `files_agent.db` |
| `databaseExportDir` | `androidAppContext.cacheDir` | Temp location for archive export |
| `remoteHostsDir` | `tmpDir/remote_hosts` | Remote host file staging |
| `preferencesDir` | `dataDir/shared_prefs` | Android SharedPreferences directory |
### Desktop Actual Values
From [Files.desktop.kt](../common/src/desktopMain/kotlin/chat/simplex/common/platform/Files.desktop.kt):
| Variable | Value | Notes |
|----------|-------|-------|
| `dataDir` | `desktopPlatform.dataPath` | XDG_DATA_HOME (Linux), AppData (Windows), Application Support (macOS) |
| `tmpDir` | `java.io.tmpdir/simplex` | System temp with `deleteOnExit` |
| `filesDir` | `dataDir/simplex_v1_files` | Flat file storage |
| `appFilesDir` | Same as `filesDir` | No subdirectory on desktop |
| `wallpapersDir` | `dataDir/simplex_v1_assets/wallpapers` | Custom wallpaper images |
| `coreTmpDir` | `dataDir/tmp` | Haskell core temp directory |
| `dbAbsolutePrefixPath` | `dataDir/simplex_v1` | Prefix: core appends `_chat.db` / `_agent.db` |
| `chatDatabaseFileName` | `"simplex_v1_chat.db"` | Full filename: `simplex_v1_chat.db` |
| `agentDatabaseFileName` | `"simplex_v1_agent.db"` | Full filename: `simplex_v1_agent.db` |
| `databaseExportDir` | Same as `tmpDir` | Temp location for archive export |
| `remoteHostsDir` | `dataDir/remote_hosts` | Remote host file staging |
| `preferencesDir` | `desktopPlatform.configPath` | Platform config directory |
### Resulting Database Paths
| Platform | Chat DB | Agent DB |
|----------|---------|----------|
| Android | `/data/data/<pkg>/files_chat.db` | `/data/data/<pkg>/files_agent.db` |
| Desktop (Linux) | `~/.local/share/simplex/simplex_v1_chat.db` | `~/.local/share/simplex/simplex_v1_agent.db` |
| Desktop (macOS) | `~/Library/Application Support/simplex/simplex_v1_chat.db` | ... |
| Desktop (Windows) | `%APPDATA%/simplex/simplex_v1_chat.db` | ... |
---
## 3. Haskell Store Modules
The Haskell core organizes database access into store modules. Kotlin code invokes these indirectly through `CC` commands. The store modules are:
| Module | Path | Responsibilities |
|--------|------|-----------------|
| `Messages.hs` | `src/Simplex/Chat/Store/Messages.hs` | Message CRUD, chat items, reactions, delivery statuses, TTL cleanup |
| `Groups.hs` | `src/Simplex/Chat/Store/Groups.hs` | Group profiles, membership, roles, invitations, group links |
| `Direct.hs` | `src/Simplex/Chat/Store/Direct.hs` | Contact management, direct connections, contact requests |
| `Files.hs` | `src/Simplex/Chat/Store/Files.hs` | File transfer metadata, XFTP state, standalone files |
| `Profiles.hs` | `src/Simplex/Chat/Store/Profiles.hs` | User profiles, display names, address book |
| `Connections.hs` | `src/Simplex/Chat/Store/Connections.hs` | SMP agent connections, pending connections, server switches |
All store operations execute within SQLite transactions managed by the Haskell core. The Kotlin layer has no direct knowledge of table schemas or SQL queries.
---
## 4. Migrations
### JNI Entry Point
Database migration is triggered by the `chatMigrateInit` external function ([Core.kt#L25](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L25)):
```kotlin
external fun chatMigrateInit(dbPath: String, dbKey: String, confirm: String): Array<Any>
```
**Parameters:**
- `dbPath` -- the `dbAbsolutePrefixPath` (core appends `_chat.db` and `_agent.db`)
- `dbKey` -- encryption passphrase (empty string = unencrypted)
- `confirm` -- migration confirmation mode: `"error"`, `"yesUp"`, or `"yesUpDown"`
**Returns:** `Array<Any>` where:
- `[0]` -- JSON string encoding a `DBMigrationResult`
- `[1]` -- `ChatCtrl` handle (Long) if migration succeeded
### Migration Flow in `initChatController`
The full initialization sequence is in [Core.kt#L62](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L62):
1. Obtain the DB encryption key from `DatabaseUtils.useDatabaseKey()`.
2. Determine the confirmation mode (default: `YesUp`; developer mode with confirm upgrades: `Error`).
3. Call `chatMigrateInit(dbAbsolutePrefixPath, dbKey, "error")` -- first attempt with `Error` to detect pending migrations.
4. Parse the result as `DBMigrationResult`.
5. If the result is `ErrorMigration` with an `Upgrade` error and confirmation allows it, re-run `chatMigrateInit` with the appropriate confirmation (`"yesUp"`).
6. If `OK`, store the `ChatCtrl` handle, set `chatDbEncrypted`, and proceed to start the chat.
7. If not `OK`, handle special case: if the `newDatabaseInitialized` preference is not set AND the database was only partially initialized (single DB file exists), remove both files and retry once.
<a id="DBMigrationResult"></a>
### DBMigrationResult
Defined in [DatabaseUtils.kt#L79](../common/src/commonMain/kotlin/chat/simplex/common/views/helpers/DatabaseUtils.kt#L79):
```kotlin
sealed class DBMigrationResult {
object OK // Migration succeeded
object InvalidConfirmation // Invalid confirmation parameter
data class ErrorNotADatabase(val dbFile: String) // File exists but is not a valid database
data class ErrorMigration(val dbFile: String, // Migration error with details
val migrationError: MigrationError)
data class ErrorSQL(val dbFile: String, // SQL error during migration
val migrationSQLError: String)
object ErrorKeychain // Keychain/keystore error
data class Unknown(val json: String) // Unparseable response
}
```
### MigrationError
```kotlin
sealed class MigrationError {
class Upgrade(val upMigrations: List<UpMigration>) // Pending forward migrations
class Downgrade(val downMigrations: List<String>) // Database is newer than app
class Error(val mtrError: MTRError) // Conflict or missing migrations
}
```
### MigrationConfirmation
```kotlin
enum class MigrationConfirmation(val value: String) {
YesUp("yesUp"), // Auto-confirm forward migrations
YesUpDown("yesUpDown"), // Auto-confirm both directions (not used in UI)
Error("error") // Report errors without running migrations
}
```
---
## 5. Database Encryption
### Encryption API
Two API functions manage database encryption, both in [SimpleXAPI.kt](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt):
| Function | Parameters | Description | Line |
|----------|-----------|-------------|------|
| `apiStorageEncryption` | `currentKey: String, newKey: String` | Change or set the database encryption passphrase | [L999](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L999) |
| `testStorageEncryption` | `key: String, ctrl: ChatCtrl?` | Test whether a given key can decrypt the database | [L1006](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L1006) |
Both delegate to the Haskell core via `CC.ApiStorageEncryption(DBEncryptionConfig)` and `CC.TestStorageEncryption(key)` respectively.
<a id="DBEncryptionConfig"></a>
`DBEncryptionConfig` ([SimpleXAPI.kt#L4166](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L4166)):
```kotlin
class DBEncryptionConfig(val currentKey: String, val newKey: String)
```
### Passphrase Storage -- CryptorInterface
The `CryptorInterface` ([Cryptor.kt](../common/src/commonMain/kotlin/chat/simplex/common/platform/Cryptor.kt)) provides platform-specific key encryption for storing the DB passphrase at rest:
```kotlin
interface CryptorInterface {
fun decryptData(data: ByteArray, iv: ByteArray, alias: String): String?
fun encryptText(text: String, alias: String): Pair<ByteArray, ByteArray>
fun deleteKey(alias: String)
}
expect val cryptor: CryptorInterface
```
### Android Implementation
[Cryptor.android.kt](../common/src/androidMain/kotlin/chat/simplex/common/platform/Cryptor.android.kt):
- Uses **Android KeyStore** (`"AndroidKeyStore"` provider)
- Algorithm: **AES/GCM/NoPadding** (128-bit authentication tag)
- Keys are hardware-backed when available
- On decryption failure with a random initial passphrase, throws to prevent overwriting
- Shows user alerts for keychain errors
```kotlin
internal class Cryptor: CryptorInterface {
private var keyStore: KeyStore = KeyStore.getInstance("AndroidKeyStore").apply { load(null) }
// AES-GCM encryption/decryption using AndroidKeyStore-managed keys
}
```
### Desktop Implementation
[Cryptor.desktop.kt](../common/src/desktopMain/kotlin/chat/simplex/common/platform/Cryptor.desktop.kt):
- **Placeholder/no-op implementation** -- data is returned as-is
- No actual encryption of the stored passphrase on desktop
- `decryptData` returns `String(data)` without decryption
- `encryptText` returns the raw bytes without encryption
```kotlin
actual val cryptor: CryptorInterface = object : CryptorInterface {
override fun decryptData(data: ByteArray, iv: ByteArray, alias: String): String? = String(data)
override fun encryptText(text: String, alias: String) = text.toByteArray() to text.toByteArray()
override fun deleteKey(alias: String) {}
}
```
### Passphrase Management
`DatabaseUtils` ([DatabaseUtils.kt](../common/src/commonMain/kotlin/chat/simplex/common/views/helpers/DatabaseUtils.kt)) provides:
- `ksDatabasePassword` -- encrypted passphrase stored in platform preferences (SharedPreferences on Android, file-based on desktop)
- `useDatabaseKey()` -- retrieves the passphrase, decrypting it via `CryptorInterface`
- `randomDatabasePassword()` -- generates a 32-byte random passphrase (Base64-encoded) for initial database creation
The flow:
1. On first launch, `randomDatabasePassword()` generates a key.
2. `CryptorInterface.encryptText()` encrypts the key for storage.
3. The encrypted (data, IV) pair is saved to preferences via `ksDatabasePassword`.
4. On subsequent launches, `ksDatabasePassword.get()` retrieves the encrypted pair, and `CryptorInterface.decryptData()` recovers the plaintext key.
5. The key is passed to `chatMigrateInit` to open the encrypted SQLite databases.
---
## 6. File Storage
### Directory Layout
Declared in [Files.kt](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt) with platform-specific implementations:
| Directory | Variable | Android Path | Desktop Path | Purpose |
|-----------|----------|-------------|--------------|---------|
| App files | `appFilesDir` | `dataDir/files/app_files` | `dataDir/simplex_v1_files` | Chat file attachments (images, videos, documents) |
| Wallpapers | `wallpapersDir` | `dataDir/files/assets/wallpapers` | `dataDir/simplex_v1_assets/wallpapers` | Custom chat wallpaper images |
| Core temp | `coreTmpDir` | `dataDir/files/temp_files` | `dataDir/tmp` | Haskell core temporary files (in-progress transfers) |
| App temp | `tmpDir` | `getDir("temp", MODE_PRIVATE)` | `java.io.tmpdir/simplex` | Application-level temporary files |
| Remote hosts | `remoteHostsDir` | `tmpDir/remote_hosts` | `dataDir/remote_hosts` | Files staged for remote host sessions |
| DB export | `databaseExportDir` | `androidAppContext.cacheDir` | Same as `tmpDir` | Temporary storage for database archive ZIP |
| Preferences | `preferencesDir` | `dataDir/shared_prefs` | `desktopPlatform.configPath` | User preferences, theme YAML |
| Migration temp | `getMigrationTempFilesDirectory()` | `dataDir/migration_temp_files` | `dataDir/migration_temp_files` | Temporary files during database migration |
### File Path Resolution
Files referenced by chat items use `CryptoFile` (optional encryption metadata + relative path). Path resolution is handled by helper functions in [Files.kt](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt):
- `getAppFilePath(fileName)` ([L81](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt#L81)) -- resolves to `appFilesDir/fileName` for local, or `remoteHostsDir/<storePath>/simplex_v1_files/fileName` for remote hosts
- `getWallpaperFilePath(fileName)` ([L91](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt#L91)) -- resolves wallpaper paths similarly
- `getLoadedFilePath(file)` ([L105](../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt#L105)) -- returns the full path if the file is downloaded and ready
### Local File Encryption
The `apiSetEncryptLocalFiles(enable)` command ([SimpleXAPI.kt#L967](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L967)) tells the Haskell core to encrypt files stored in `appFilesDir`. When enabled, files are written as `CryptoFile` with a random AES key and nonce. The JNI functions `chatEncryptFile` and `chatDecryptFile` ([Core.kt#L39-L40](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L39)) handle the actual crypto operations.
---
## 7. Export & Import
### API Functions
| Function | CC Command | CR Response | Line |
|----------|-----------|-------------|------|
| `apiExportArchive(config)` | `CC.ApiExportArchive(config)` | `CR.ArchiveExported(archiveErrors)` | [SimpleXAPI.kt#L981](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L981) |
| `apiImportArchive(config)` | `CC.ApiImportArchive(config)` | `CR.ArchiveImported(archiveErrors)` | [SimpleXAPI.kt#L987](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L987) |
| `apiDeleteStorage()` | `CC.ApiDeleteStorage()` | `CR.CmdOk` | [SimpleXAPI.kt#L993](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L993) |
### ArchiveConfig
Defined at [SimpleXAPI.kt#L4162](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L4162):
```kotlin
class ArchiveConfig(
val archivePath: String, // Full path to the ZIP archive
val disableCompression: Boolean?, // Skip compression for speed
val parentTempDirectory: String? // Temp directory for extraction
)
```
### Export Flow
1. UI constructs an `ArchiveConfig` with a path under `databaseExportDir`.
2. Calls `apiExportArchive(config)` which sends `CC.ApiExportArchive` to the Haskell core.
3. The core creates a ZIP containing both `_chat.db` and `_agent.db` (and optionally files).
4. Returns `CR.ArchiveExported` with a list of `ArchiveError` (non-fatal issues during export).
5. UI offers the archive file for sharing/saving.
### Import Flow
1. User selects an archive file.
2. UI copies it to a temp location and constructs an `ArchiveConfig`.
3. Calls `apiImportArchive(config)` which sends `CC.ApiImportArchive` to the Haskell core.
4. The core extracts and replaces both databases.
5. Returns `CR.ArchiveImported` with a list of `ArchiveError` (non-fatal issues during import).
6. UI triggers re-initialization via `initChatController`.
<a id="ArchiveError"></a>
### ArchiveError
Defined at [SimpleXAPI.kt#L7658](../common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt#L7658):
```kotlin
sealed class ArchiveError {
class ArchiveErrorImport(val importError: String) // General import error
class ArchiveErrorFile(val file: String, val fileError: String) // Per-file error
}
```
### Delete Storage
`apiDeleteStorage()` removes both database files entirely. This is used during account deletion or database reset operations. After calling this, `initChatController` must be called to create fresh databases.
---
## 8. Source Files
| File | Purpose | Path |
|------|---------|------|
| SimpleXAPI.kt | API functions: encryption, export/import, storage commands | `common/src/commonMain/kotlin/chat/simplex/common/model/SimpleXAPI.kt` |
| Core.kt | JNI externals (`chatMigrateInit`, `chatEncryptFile`, etc.), `initChatController` | `common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt` |
| Files.kt | Platform-expect file/directory path declarations | `common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt` |
| Files.android.kt | Android actual paths (dataDir, appFilesDir, etc.) | `common/src/androidMain/kotlin/chat/simplex/common/platform/Files.android.kt` |
| Files.desktop.kt | Desktop actual paths (XDG/AppData, etc.) | `common/src/desktopMain/kotlin/chat/simplex/common/platform/Files.desktop.kt` |
| Cryptor.kt | Platform-expect encryption interface for passphrase storage | `common/src/commonMain/kotlin/chat/simplex/common/platform/Cryptor.kt` |
| Cryptor.android.kt | Android: AES-GCM via AndroidKeyStore | `common/src/androidMain/kotlin/chat/simplex/common/platform/Cryptor.android.kt` |
| Cryptor.desktop.kt | Desktop: placeholder (no-op) implementation | `common/src/desktopMain/kotlin/chat/simplex/common/platform/Cryptor.desktop.kt` |
| DatabaseUtils.kt | `DBMigrationResult`, `MigrationError`, `MigrationConfirmation`, passphrase helpers | `common/src/commonMain/kotlin/chat/simplex/common/views/helpers/DatabaseUtils.kt` |
| Messages.hs | Haskell store: message CRUD, reactions, delivery | `src/Simplex/Chat/Store/Messages.hs` |
| Groups.hs | Haskell store: groups, membership, roles | `src/Simplex/Chat/Store/Groups.hs` |
| Direct.hs | Haskell store: contacts, direct connections | `src/Simplex/Chat/Store/Direct.hs` |
| Files.hs | Haskell store: file transfer metadata | `src/Simplex/Chat/Store/Files.hs` |
| Profiles.hs | Haskell store: user profiles | `src/Simplex/Chat/Store/Profiles.hs` |
| Connections.hs | Haskell store: SMP agent connections | `src/Simplex/Chat/Store/Connections.hs` |
All Kotlin paths are relative to `apps/multiplatform/`. All Haskell paths are relative to the repository root.
Reference in New Issue View Git Blame Copy Permalink