40 Commits

Author SHA1 Message Date
Evgeny
87e8a10f1e core: use strict tables (#6535)
* core: use strict tables

* fix field types

* change encodings to match schema types; migrate sqlite tables to strict mode

* stabilize postgres client tests, remove slow handshake tests

* update simplexmq

* fix test

* change call_state type to text

* fix directory service queries

* update local_alias for existing schemas

* change types before strict
2026-01-05 08:53:26 +00:00
spaced4ndy
fe4ff8993d core: finalize introductions -> member relations vector stage 2 migration (drop introductions) (#6490)
* core: finalize introductions -> member relations vector stage 2 migration (drop introductions)

* remove comment

* skip down migration check

* fix

* plans

* postgres schema

* skip down migration comparison

* do not drop group_member_intros table, rename migrations

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-12-26 13:29:48 +00:00
spaced4ndy
f76e994034 core: forward based on relations vector (#6464)
* core: forward based on relations vector wip

* fix introductions

* fix forwarding tests

* fix forwarding inside support scope

* fix deduplication test

* fix more tests

* plans, api

* live migration wip

* enable tests

* member locks

* api

* plans

* fix for postgres

* fix for postgres

* rename predicate

* rename predicate

* optimize

* refactor

* fix

* check

* move part of migration to sql

* plans

* core: preserve detailed information in relation vectors (#6484)

* core: relations vector live migrations; stage 2 migration sql (#6472)

* rework forwarding in support scope

* move operations inside transactions

* set_member_vector_new_relation function

* read vector ad-hoc

* partition in transaction

* fix postgres

* postgres schema

* api

* plans

* remove comment

* lock before migration computation

* refactor

* simplify set relations

* retreive only support scope members

* fix

* refactor

* fix comment

* enable tests

* 1 second

* for update

* locks

* fix mask

* plans

* fix

* postgres

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-12-09 14:48:32 +00:00
spaced4ndy
c782e6762b feature branch: optimize subscriptions (#6297)
* core: remove contact merge, obsolete group code (#6292)

* core: remove file connections (#6298)

* core: rename migration

* corre: remove unused contacts merged event (#6308)

* core: optimize subscriptions (#6304)

* core: remove conn ids map from subscriptions (#6309)

* rename migration (#6316)

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-09-28 21:31:59 +01:00
Evgeny
015d5de364 directory: move all storage to the database, optimize performance (#6287)
* core: add custom indexed columns to groups and contacts

* directory: use database (TODO search/listing)

* triggers to maintain current member count

* update simplexmq, fix tests, use summary from GroupInfo

* fix all directory tests

* remove acceptance fields from group reg

* enable all tests

* clean up

* postgres migrations, fixes

* query plans

* use function in postgres triggers, improve sqlite query

* fix export/import

* update schema

* prevent admins from promoting groups when approving

* update listing every 5 minutes
2025-09-28 15:11:49 +01:00
Evgeny
fa9bc1537c cli: option to disable backup or change backup directory on migration (#6267)
* cli: option to disable backup or change backup directory on migration

* update nix

* update simplexmq
2025-09-14 14:41:50 +01:00
spaced4ndy
d69511bcad core: improve indexes (#6095)
* core: improve indexes

* update plans

* queries

* postgres migration

* optimize query

* lint

* plans

* add indexes

* schema

* plans

* postgres schema

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-07-24 23:36:08 +01:00
Evgeny
b2de37a9fb core: member acceptance (#5678)
* core: member acceptance

* migration

* move hook

* core: support sending direct messages to members (#5680)

* fix compilation, todos

* fix test

* predicates

* comment

* extend hook

* wip

* wip

* wip

* wip

* fix test

* mute output

* schema

* better query

* plans

* fix test

* directory

* captcha

* captcha works

* remove column, add UI types and group status icon

* fix test

* query plans

* exclude messages of pending members from history

* commands for filter settings

* core: separately delete pending approval members; other apis validation (#5699)

* accepted status

* send captcha messages as replies

* fix blocked words

* simpler filter info

* info about /filter and /role after group registration

* update query plans

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
2025-03-03 18:57:29 +00:00
spaced4ndy
9e000d6bce core: optimize group deletion (#5565)
* core: optimize group deletion

* withFastStore

* fix indexes

* updated plans

* remove prints

* remove print

* undo diff

* core: optimize group delete - delayed group cleanup, delete unused contacts before deleting group (#5579)

* core: delete unused group contacts, don't create new ones

* remove from exceptions

* plans

* fix tests

* remove fixtures

* update plans

* update plans

* fix test

* remove unused functino

* update plans

* remove withFastStore

* core: time group deletion (#5596)

* core: time group deletion

* queries

* works, test fails

* fix

* update plans

* update migration, queries

* not null

* remove deleted

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* cleanup

* remove unused field

* fix

* fix

* plans

* fix plan save

* plans

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-01-31 18:47:59 +04:00
spaced4ndy
5d18a49726 core: delete unused group contacts, don't create new ones (#5590)
* core: delete unused group contacts, don't create new ones

* remove from exceptions

* plans

* fix tests

* remove fixtures

* update plans

* update migration

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-01-31 10:32:07 +04:00
Evgeny
621b291da1 core: member mentions, types and rfc (#5555)
* core: member mentions, types and rfc

* update

* update rfc

* save/get mentions (WIP)

* markdown

* store received mentions and userMention flag

* sent mentions

* update message with mentions

* db queries

* CLI mentions, test passes

* use maps for mentions

* tests

* comment

* save mentions on sent messages

* postresql schema

* refactor

* M.empty

* include both displayName and localAlias into MentionedMemberInfo

* fix saving sent mentions

* include mentions in previews

* update plans
2025-01-29 13:04:48 +00:00
spaced4ndy
d86e6b35be test: track agent query plans (#5571) 2025-01-24 17:49:31 +04:00
Evgeny
f3664619ec test: track query plans (#5566)
* test: track query plans

* all query plans

* fix postgres build
2025-01-24 09:44:53 +00:00
spaced4ndy
20fa30eacc core: Mobile.hs postgres interface (#5545)
* core: Mobile.hs postgres interface

* sqlite

* fix

* errors

* postgres

* rename

* rename, refactor

* merge files

* rename

* update simplexmq

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-01-20 17:41:48 +04:00
spaced4ndy
e05a35e26e core: support postgres backend (#5403)
* postgres: modules structure (#5401)

* postgres: schema, field conversions (#5430)

* postgres: rework chat list pagination query (#5441)

* prepare cabal for merge

* restore cabal changes

* simplexmq

* postgres: implementation wip (tests don't pass) (#5481)

* restore ios file

* postgres: implementation - tests pass (#5487)

* refactor DB options

* refactor

* line

* style

* style

* refactor

* $

* update simplexmq

* constraintError

* handleDBErrors

* fix

* remove param

* Ok

* case

* case

* case

* comment

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2025-01-10 11:27:29 +00:00
Evgeny
206f7898c3 cli: option to disable vacuum on migration (#5446)
* cli: option to disable vacuum on migration

* update simplexmq

* mobile options

* use option in test
2024-12-28 22:14:06 +00:00
Evgeny
d37d309f85 core: update simplemq (with PostgreSQL support) (#5444) 2024-12-28 12:35:34 +00:00
spaced4ndy
d912fe07a1 core: fix pagination indexes (#5241) 2024-11-25 18:51:49 +04:00
Diogo
4d82209a3a core: pagination API to load items around defined or the earliest unread item (#5100)
* core: auto increment chat item ids (#5088)

* core: auto increment chat item ids

* file name

* down name

* update schema

* ignore down migration on schema dump test

* fix testDirectMessageDelete test

* fix testNotes test

* core: initial api support for items around a given item (#5092)

* core: initial api support for items around a given item

* implementation and tests for local messages

* pass entities down

* unused

* getAllChatItems implementation and tests

* pagination for getting chat and tests

* remove unused import

* group implementation and tests

* refactor

* order by created at for local and direct chats

* core: initial landing api for chat and gaps  (#5104)

* initial work on initial param for loading chat

* support for initial

* controller parse

* fixed sqls

* refactor names

* fix ChatLandingSection serialized type

* total accuracy on landing section

* descriptive view message

* foldr

* refactor to make landingSection reusable

* refactor: use foldr everywhere

* propagate search

* Revert "propagate search"

This reverts commit 01611fd719.

* throw when search is sent for initial

* gap size wip (needs testing)

* final

* remove order by

* remove index

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

* core: fix initial api latest chat items ordering (#5151)

* core: fix one item missing from latest in initial and wrong check (#5153)

* core: fix one item missing from latest in initial and wrong check

* final fixes and tests

* clearer tests

* core: remove gaps and make sure page size is always the same (#5163)

* remove gaps

* consistent pagination size

* proper fix and around fix too

* optimize

* refactor

* core: simplify pagination

* core: first unread queries (#5174)

* core: pagination nav info (#5175)

* core: pagination nav info

* wip

* rework

* rework

* group, local

* fix

* rename

* fix tests

* just

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2024-11-14 08:34:25 +00:00
spaced4ndy
12423f4afa core: test db indexes (#4999) 2024-10-09 15:15:58 +04:00
spaced4ndy
366b58ead5 core: don't save async commands records for ack (#3904) 2024-03-28 19:52:06 +04:00
spaced4ndy
12d1ada25e core: support batch sending in groups, batch introductions; send recent message history to new members (#3519)
* core: batch send stubs, comments

* multiple events in ChatMessage and supporting types

* Revert "multiple events in ChatMessage and supporting types"

This reverts commit 9b239b26ba.

* schema, refactor group processing for batched messages

* encoding, refactor processing

* refactor code to work with updated schema

* encoding, remove instances

* wip

* implement batching

* batch introductions

* wip

* collect and send message history

* missing new line

* rename

* test

* rework to build history via chat items

* refactor, tests

* correctly set member version range, dont include deleted items

* tests

* fix disappearing messages

* check number of errors

* comment

* check size in encodeChatMessage

* fix - don't check msg size for binary

* use builder

* rename

* rename

* rework batching

* lazy msg body

* use withStoreBatch

* refactor

* reverse batches

* comment

* possibly fix builder for single msg

* refactor batcher

* refactor

* dont repopulate msg_deliveries on down migration

* EncodedChatMessage type

* remove type

* batcher tests

* add tests

* group history preference

* test group link

* fix tests

* fix for random update

* add test testImageFitsSingleBatch

* refactor

* rename function

* refactor

* mconcat

* rename feature

* catch error on each batch

* refactor file inv retrieval

* refactor gathering item forward events

* refactor message batching

* unite migrations

* move files

* refactor

* Revert "unite migrations"

This reverts commit 0be7a3117a.

* refactor splitFileDescr

* improve tests

* Revert "dont repopulate msg_deliveries on down migration"

This reverts commit 2944c1cc28.

* fix down migration

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-12-23 13:07:23 +00:00
Evgeny Poberezkin
d3059afc99 ios, core: better notifications processing to avoid contention for database (#3485)
* core: forward notifications about message processing (for iOS notifications)

* simplexmq

* the option to keep database key, to allow re-opening the database

* export new init with keepKey and reopen DB api

* stop remote ctrl when suspending chat

* ios: close/re-open db on suspend/activate

* allow activating chat without restoring (for NSE)

* update NSE to suspend/activate (does not work)

* simplexmq

* suspend chat and close database when last notification in the process is processed

* stop reading notifications on message markers

* replace async stream with cancellable concurrent queue

* better synchronization of app and NSE

* remove outside of task

* remove unused var

* whitespace

* more debug logging, handle cancelled read after dequeue

* comments

* more comments
2023-12-09 21:59:40 +00:00
spaced4ndy
87d84cfccc core: filter connection plan query results by user_id (#3251) 2023-10-21 19:13:32 +04:00
spaced4ndy
1928256b09 core: connect existing contacts to new members when profile matches, enable skipping direct connections in groups (#3056)
* core: test group members are assigned different LDNs in group when direct connections aren't created

* disable test output

* core: connect existing contacts to new members when profile matches (#3059)

* core: connect existing contacts to new members when profile matches

* fix migration

* progress

* xInfoProbeOk for member

* fix tests

* add test

* fix tests

* tests

* remove deleteSentProbe

* remove deleteContactProfile_

* views

* don't check connections in deleteUnusedProfile_

* Revert "don't check connections in deleteUnusedProfile_"

This reverts commit 2016a0efde.

* fix test

* core: update member merge

* update saved schema

* fix queries and tests

* rename tables to original names

* remove index, corrections

* update schema dump

---------

Co-authored-by: spaced4ndy <8711996+spaced4ndy@users.noreply.github.com>

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>
2023-09-19 21:26:03 +01:00
Evgeny Poberezkin
75d11c2b4f core: update simplexmq (fix repeated HELLO message) (#2962)
* core: update simplexmq (fix repeated HELLO message)

* add callstack to tests
2023-08-21 21:45:16 +01:00
Evgeny Poberezkin
113669ac16 core: track slow SQL queries (#2904)
* core: track slow SQL queries

* fixes

* update simplexmq
2023-08-12 18:27:10 +01:00
Evgeny Poberezkin
38dc14f041 core: update simplexmq (revert DB busy error handling) 2023-08-09 16:45:31 +01:00
spaced4ndy
34c2303ef1 core: update simplexmq (db error busy treatments) (#2676) 2023-07-11 20:57:14 +04:00
spaced4ndy
128883b8a3 core: improve queries performance; delay first chat item expiration cycle on start (#2521) 2023-05-29 15:18:22 +04:00
spaced4ndy
54fc052e47 core: remove msg_delivery_events unique constraint (recreates table); cleanup old messages (#2376) 2023-05-05 13:49:09 +04:00
Evgeny Poberezkin
0c3dc8a6e9 core: add down migrations and fix test 2023-03-27 19:39:22 +01:00
Evgeny Poberezkin
c96ba30018 core: support down migrations to allow reverting to the previous version (#2072)
* core: support down migrations to allow reverting to the previous version

* update schema

* update simplexmq

* rename errors

* remove unused functions

* migration UI, test migration

* update migration UI

* return current migrations in CRVersionInfo

* update simplexmq

* test down migrations

* cleanup ios

* show migrations in log
2023-03-27 18:34:48 +01:00
Evgeny Poberezkin
a1ed0a84b8 core: use port 7001 for test server (#1857)
* core: use port 7001 for test server

* enable only failing tests

* start/stop server for every test

* log message that failed to parse

* stop chat synchronously

* print call stack

* add HasCallStack

* increase test timeout

* add call stacks

* more call stacks

* fix test

* disable failing test

* add delay between the tests

* make delay more visible

* remove change in error message

* reduce test delay, increase timeout

* increase delay between the tests

* run each test with a database in a different folder

* folder name

* refactor

* update nix file, more stacks
2023-01-31 11:07:48 +00:00
Evgeny Poberezkin
2b5e3a9459 core: C API to migrate and check database (#1008)
* core: C API to migrate and check database

* update simplexmq
2022-09-02 16:38:41 +01:00
Evgeny Poberezkin
02ca7234fb use SQLCipher (#981)
* use SQLCipher

* pass encryption key via CLI options

* update dependencies to use git

* add CONTRIBUTING.md

* move flag, enable build in sqlcipher branch

* update dependencies
2022-08-30 12:49:07 +01:00
Evgeny Poberezkin
687e3be9ac iOS: update call invitations when exiting background (#771)
* core: communicate call invitations state between NSE and app via db

* enable tests

* delete calls, encoding

* load calls on start

* remove line

* remove table alias

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
2022-07-04 14:15:25 +04:00
Evgeny Poberezkin
41c9742b0d core: auto-reply message for user contact addresses (#755)
* core: auto-reply message for user contact addresses

* terminal: show auto accept status and message

* test
2022-06-27 19:41:25 +01:00
Evgeny Poberezkin
c62d99ab97 core: remove connection pool (#738)
* core: remove connection pool

* remove local ref from cabal.project

* update simplexmq

* log test

* fix test
2022-06-16 20:00:51 +01:00
JRoberts
de64f3a1a0 tests: maintain schema dump (#505) 2022-04-05 12:44:22 +04:00